2023-09-29T23:20:48.265 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-29T23:20:48.265 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586 --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --owner scheduled_yuriw@teuthology --verbose -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586/orig.config.yaml 2023-09-29T23:20:48.310 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-29T23:20:48.387 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7406586' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPvPERG99uKqCE5cUKrNqI7eYuJHBXWPBlFgCuZx0NmMh3J9k7ndvs1VBB1bF9I+97RdCkmCFnsnYhpADHvH5Gs= smithi135.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPUUjOeXF2ooqKQhpZAj7iSD5ij/F0ffRzx+l37fx47VQbEGqYBo7TgFF0BSKZOzRhalCw7yOEhzZs87TAF2ueo= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-29T23:20:48.387 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-29T23:20:48.388 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-29T23:20:48.388 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-29T23:20:48.390 INFO:teuthology.task.internal:Checking packages... 2023-09-29T23:20:48.412 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-29T23:20:48.413 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-29T23:20:48.413 INFO:teuthology.packaging:ref: None 2023-09-29T23:20:48.413 INFO:teuthology.packaging:tag: None 2023-09-29T23:20:48.413 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-29T23:20:48.413 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T23:20:48.413 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-29T23:20:48.556 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165.g93dce5e1 2023-09-29T23:20:48.556 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-29T23:20:48.564 INFO:teuthology.task.internal:no buildpackages task found 2023-09-29T23:20:48.564 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-29T23:20:48.642 INFO:teuthology.task.internal:Saving configuration 2023-09-29T23:20:48.661 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-29T23:20:48.669 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-29T23:20:48.694 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 23:15:47.011478', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPvPERG99uKqCE5cUKrNqI7eYuJHBXWPBlFgCuZx0NmMh3J9k7ndvs1VBB1bF9I+97RdCkmCFnsnYhpADHvH5Gs='} 2023-09-29T23:20:48.717 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 23:15:47.013443', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPUUjOeXF2ooqKQhpZAj7iSD5ij/F0ffRzx+l37fx47VQbEGqYBo7TgFF0BSKZOzRhalCw7yOEhzZs87TAF2ueo='} 2023-09-29T23:20:48.717 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-29T23:20:48.725 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-09-29T23:20:48.725 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-09-29T23:20:48.725 INFO:teuthology.run_tasks:Running task console_log... 2023-09-29T23:20:48.798 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f550e666ee0>, signals=[15]) 2023-09-29T23:20:48.798 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-29T23:20:48.806 INFO:teuthology.task.internal:Opening connections... 2023-09-29T23:20:48.806 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2023-09-29T23:20:48.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:20:48.878 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi135.front.sepia.ceph.com 2023-09-29T23:20:48.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:20:48.947 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-29T23:20:48.953 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2023-09-29T23:20:48.977 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2023-09-29T23:20:48.978 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2023-09-29T23:20:49.039 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2023-09-29T23:20:49.039 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="8" 2023-09-29T23:20:49.039 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2023-09-29T23:20:49.039 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2023-09-29T23:20:49.040 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="8" 2023-09-29T23:20:49.040 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el8" 2023-09-29T23:20:49.040 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-29T23:20:49.040 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2023-09-29T23:20:49.040 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-29T23:20:49.040 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2023-09-29T23:20:49.040 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-29T23:20:49.040 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-29T23:20:49.040 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-29T23:20:49.041 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2023-09-29T23:20:49.060 DEBUG:teuthology.orchestra.run.smithi135:> uname -m 2023-09-29T23:20:49.083 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2023-09-29T23:20:49.083 DEBUG:teuthology.orchestra.run.smithi135:> cat /etc/os-release 2023-09-29T23:20:49.144 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="CentOS Stream" 2023-09-29T23:20:49.145 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="8" 2023-09-29T23:20:49.145 INFO:teuthology.orchestra.run.smithi135.stdout:ID="centos" 2023-09-29T23:20:49.145 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE="rhel fedora" 2023-09-29T23:20:49.145 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="8" 2023-09-29T23:20:49.145 INFO:teuthology.orchestra.run.smithi135.stdout:PLATFORM_ID="platform:el8" 2023-09-29T23:20:49.145 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-29T23:20:49.145 INFO:teuthology.orchestra.run.smithi135.stdout:ANSI_COLOR="0;31" 2023-09-29T23:20:49.145 INFO:teuthology.orchestra.run.smithi135.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-29T23:20:49.145 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://centos.org/" 2023-09-29T23:20:49.145 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-29T23:20:49.145 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-29T23:20:49.146 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-29T23:20:49.146 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2023-09-29T23:20:49.168 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-29T23:20:49.178 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-29T23:20:49.184 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-29T23:20:49.185 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2023-09-29T23:20:49.187 DEBUG:teuthology.orchestra.run.smithi135:> test '!' -e /home/ubuntu/cephtest 2023-09-29T23:20:49.208 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-29T23:20:49.215 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-29T23:20:49.215 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2023-09-29T23:20:49.249 DEBUG:teuthology.orchestra.run.smithi135:> test -z $(ls -A /var/lib/ceph) 2023-09-29T23:20:49.281 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-29T23:20:49.328 INFO:teuthology.run_tasks:Running task kernel... 2023-09-29T23:20:49.343 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-29T23:20:49.344 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-29T23:20:49.344 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-29T23:20:49.345 DEBUG:teuthology.orchestra.run.smithi135:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-29T23:20:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:20:49.365 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2023-09-29T23:20:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:20:49.366 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2023-09-29T23:20:49.426 INFO:teuthology.orchestra.run.smithi040.stdout:4.18.0-489.el8.x86_64 2023-09-29T23:20:49.426 INFO:teuthology.task.kernel:Running kernel on smithi040: 4.18.0-489.el8.x86_64 2023-09-29T23:20:49.426 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum install -y kernel 2023-09-29T23:20:49.427 INFO:teuthology.orchestra.run.smithi135.stdout:4.18.0-489.el8.x86_64 2023-09-29T23:20:49.428 INFO:teuthology.task.kernel:Running kernel on smithi135: 4.18.0-489.el8.x86_64 2023-09-29T23:20:49.428 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum install -y kernel 2023-09-29T23:20:53.580 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - AppStream 20 MB/s | 33 MB 00:01 2023-09-29T23:20:53.835 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - AppStream 20 MB/s | 33 MB 00:01 2023-09-29T23:21:11.607 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - BaseOS 4.1 MB/s | 48 MB 00:11 2023-09-29T23:21:12.078 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - BaseOS 4.0 MB/s | 48 MB 00:12 2023-09-29T23:21:19.084 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - Extras 28 kB/s | 18 kB 00:00 2023-09-29T23:21:19.753 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - Extras 45 kB/s | 18 kB 00:00 2023-09-29T23:21:19.818 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2023-09-29T23:21:20.137 INFO:teuthology.orchestra.run.smithi135.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-09-29T23:21:20.445 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2023-09-29T23:21:20.738 INFO:teuthology.orchestra.run.smithi040.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-09-29T23:21:22.016 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 9.2 MB/s | 16 MB 00:01 2023-09-29T23:21:22.856 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 16 MB 00:01 2023-09-29T23:21:25.763 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 475 kB/s | 24 kB 00:00 2023-09-29T23:21:26.948 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 428 kB/s | 24 kB 00:00 2023-09-29T23:21:29.273 INFO:teuthology.orchestra.run.smithi135.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-29T23:21:29.273 INFO:teuthology.orchestra.run.smithi135.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-29T23:21:29.528 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-29T23:21:29.529 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-29T23:21:29.529 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2023-09-29T23:21:29.529 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-29T23:21:29.529 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-29T23:21:29.529 INFO:teuthology.orchestra.run.smithi135.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-29T23:21:29.530 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-09-29T23:21:29.530 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-29T23:21:29.530 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-29T23:21:29.530 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-29T23:21:29.530 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-29T23:21:29.530 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-29T23:21:29.530 INFO:teuthology.orchestra.run.smithi135.stdout:Install 3 Packages 2023-09-29T23:21:29.530 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-29T23:21:29.531 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 88 M 2023-09-29T23:21:29.531 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 96 M 2023-09-29T23:21:29.531 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-29T23:21:30.680 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-29T23:21:30.681 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-29T23:21:30.947 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-29T23:21:30.948 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-29T23:21:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2023-09-29T23:21:30.949 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-29T23:21:30.949 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-29T23:21:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-29T23:21:30.949 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-09-29T23:21:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-29T23:21:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-29T23:21:30.949 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-29T23:21:30.949 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-29T23:21:30.950 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-29T23:21:30.950 INFO:teuthology.orchestra.run.smithi040.stdout:Install 3 Packages 2023-09-29T23:21:30.950 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-29T23:21:30.950 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 88 M 2023-09-29T23:21:30.950 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 96 M 2023-09-29T23:21:30.950 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-29T23:21:34.998 INFO:teuthology.orchestra.run.smithi135.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 1.9 MB/s | 10 MB 00:05 2023-09-29T23:21:36.015 INFO:teuthology.orchestra.run.smithi040.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 2.1 MB/s | 10 MB 00:04 2023-09-29T23:21:45.347 INFO:teuthology.orchestra.run.smithi135.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 2.3 MB/s | 35 MB 00:15 2023-09-29T23:21:47.572 INFO:teuthology.orchestra.run.smithi040.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 2.2 MB/s | 35 MB 00:16 2023-09-29T23:21:48.680 INFO:teuthology.orchestra.run.smithi135.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 2.2 MB/s | 43 MB 00:18 2023-09-29T23:21:48.680 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:21:48.681 INFO:teuthology.orchestra.run.smithi135.stdout:Total 4.6 MB/s | 88 MB 00:19 2023-09-29T23:21:49.346 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-29T23:21:49.779 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-29T23:21:49.780 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-29T23:21:50.614 INFO:teuthology.orchestra.run.smithi040.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 2.2 MB/s | 43 MB 00:19 2023-09-29T23:21:50.614 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-29T23:21:50.614 INFO:teuthology.orchestra.run.smithi040.stdout:Total 4.5 MB/s | 88 MB 00:19 2023-09-29T23:21:51.034 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-29T23:21:51.055 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-29T23:21:51.314 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-29T23:21:51.762 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-29T23:21:51.762 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-29T23:21:53.068 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-29T23:21:53.089 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-29T23:21:53.828 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-29T23:21:54.322 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-29T23:21:55.981 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-29T23:21:56.549 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-29T23:21:57.889 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-29T23:21:58.288 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-29T23:22:00.248 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-29T23:22:00.648 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-29T23:22:01.674 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-29T23:22:01.959 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-29T23:22:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-29T23:22:04.567 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-29T23:22:23.661 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-29T23:22:23.685 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-29T23:22:24.036 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-29T23:22:24.036 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-29T23:22:24.036 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-29T23:22:24.786 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-29T23:22:24.786 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-29T23:22:24.786 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-29T23:22:24.786 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-29T23:22:24.786 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-29T23:22:24.786 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-29T23:22:24.786 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-29T23:22:24.925 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T23:22:25.161 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-29T23:22:25.161 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-29T23:22:25.161 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-29T23:22:25.161 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-29T23:22:25.161 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-29T23:22:25.161 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-09-29T23:22:25.162 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi135.front.sepia.ceph.com, path=None, version=distro) 2023-09-29T23:22:25.162 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T23:22:25.370 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-29T23:22:25.370 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-29T23:22:25.371 DEBUG:teuthology.orchestra.run.smithi135:> sudo rpm -qi grub2-tools 2023-09-29T23:22:25.404 INFO:teuthology.orchestra.run.smithi135.stdout:Name : grub2-tools 2023-09-29T23:22:25.404 INFO:teuthology.orchestra.run.smithi135.stdout:Epoch : 1 2023-09-29T23:22:25.404 INFO:teuthology.orchestra.run.smithi135.stdout:Version : 2.02 2023-09-29T23:22:25.404 INFO:teuthology.orchestra.run.smithi135.stdout:Release : 129.el8 2023-09-29T23:22:25.404 INFO:teuthology.orchestra.run.smithi135.stdout:Architecture: x86_64 2023-09-29T23:22:25.404 INFO:teuthology.orchestra.run.smithi135.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-29T23:22:25.405 INFO:teuthology.orchestra.run.smithi135.stdout:Group : System Environment/Base 2023-09-29T23:22:25.405 INFO:teuthology.orchestra.run.smithi135.stdout:Size : 9265586 2023-09-29T23:22:25.405 INFO:teuthology.orchestra.run.smithi135.stdout:License : GPLv3+ 2023-09-29T23:22:25.405 INFO:teuthology.orchestra.run.smithi135.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-29T23:22:25.405 INFO:teuthology.orchestra.run.smithi135.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-29T23:22:25.405 INFO:teuthology.orchestra.run.smithi135.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-29T23:22:25.405 INFO:teuthology.orchestra.run.smithi135.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-29T23:22:25.405 INFO:teuthology.orchestra.run.smithi135.stdout:Relocations : (not relocatable) 2023-09-29T23:22:25.405 INFO:teuthology.orchestra.run.smithi135.stdout:Packager : CentOS BuildSystem 2023-09-29T23:22:25.405 INFO:teuthology.orchestra.run.smithi135.stdout:Vendor : CentOS 2023-09-29T23:22:25.405 INFO:teuthology.orchestra.run.smithi135.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-29T23:22:25.405 INFO:teuthology.orchestra.run.smithi135.stdout:Summary : Support tools for GRUB. 2023-09-29T23:22:25.406 INFO:teuthology.orchestra.run.smithi135.stdout:Description : 2023-09-29T23:22:25.406 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-29T23:22:25.406 INFO:teuthology.orchestra.run.smithi135.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-29T23:22:25.406 INFO:teuthology.orchestra.run.smithi135.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-29T23:22:25.406 INFO:teuthology.orchestra.run.smithi135.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-29T23:22:25.406 INFO:teuthology.orchestra.run.smithi135.stdout:hardware devices. 2023-09-29T23:22:25.406 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-29T23:22:25.406 INFO:teuthology.orchestra.run.smithi135.stdout:This subpackage provides tools for support of all platforms. 2023-09-29T23:22:25.407 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-29T23:22:25.407 INFO:teuthology.task.kernel:Updating grub on smithi135 to boot 4.18.0-514.el8.x86_64 2023-09-29T23:22:25.407 DEBUG:teuthology.orchestra.run.smithi135:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-29T23:22:27.175 INFO:teuthology.orchestra.run.smithi135.stderr:Generating grub configuration file ... 2023-09-29T23:22:27.720 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-29T23:22:27.745 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-29T23:22:28.105 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-29T23:22:28.105 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-29T23:22:28.106 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-29T23:22:28.865 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-29T23:22:28.866 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-29T23:22:28.866 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-29T23:22:28.866 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-29T23:22:28.866 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-29T23:22:28.866 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-29T23:22:28.866 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-29T23:22:29.081 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T23:22:29.292 INFO:teuthology.orchestra.run.smithi135.stderr:done 2023-09-29T23:22:29.294 DEBUG:teuthology.orchestra.run.smithi135:> mktemp 2023-09-29T23:22:29.303 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-29T23:22:29.304 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-29T23:22:29.304 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-29T23:22:29.305 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-29T23:22:29.305 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-29T23:22:29.305 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-29T23:22:29.305 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=None, version=distro) 2023-09-29T23:22:29.305 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2023-09-29T23:22:29.318 INFO:teuthology.orchestra.run.smithi135.stdout:/tmp/tmp.KC4ckFP01a 2023-09-29T23:22:29.318 DEBUG:teuthology.orchestra.run.smithi135:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KC4ckFP01a 2023-09-29T23:22:29.403 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0666 /tmp/tmp.KC4ckFP01a 2023-09-29T23:22:29.507 DEBUG:teuthology.orchestra.remote:smithi135:/tmp/tmp.KC4ckFP01a is 6KB 2023-09-29T23:22:29.523 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-29T23:22:29.523 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-29T23:22:29.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo rpm -qi grub2-tools 2023-09-29T23:22:29.556 DEBUG:teuthology.orchestra.run.smithi135:> rm -fr /tmp/tmp.KC4ckFP01a 2023-09-29T23:22:29.558 INFO:teuthology.orchestra.run.smithi040.stdout:Name : grub2-tools 2023-09-29T23:22:29.558 INFO:teuthology.orchestra.run.smithi040.stdout:Epoch : 1 2023-09-29T23:22:29.558 INFO:teuthology.orchestra.run.smithi040.stdout:Version : 2.02 2023-09-29T23:22:29.558 INFO:teuthology.orchestra.run.smithi040.stdout:Release : 129.el8 2023-09-29T23:22:29.558 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: x86_64 2023-09-29T23:22:29.558 INFO:teuthology.orchestra.run.smithi040.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-29T23:22:29.558 INFO:teuthology.orchestra.run.smithi040.stdout:Group : System Environment/Base 2023-09-29T23:22:29.558 INFO:teuthology.orchestra.run.smithi040.stdout:Size : 9265586 2023-09-29T23:22:29.559 INFO:teuthology.orchestra.run.smithi040.stdout:License : GPLv3+ 2023-09-29T23:22:29.559 INFO:teuthology.orchestra.run.smithi040.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-29T23:22:29.559 INFO:teuthology.orchestra.run.smithi040.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-29T23:22:29.559 INFO:teuthology.orchestra.run.smithi040.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-29T23:22:29.559 INFO:teuthology.orchestra.run.smithi040.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-29T23:22:29.559 INFO:teuthology.orchestra.run.smithi040.stdout:Relocations : (not relocatable) 2023-09-29T23:22:29.559 INFO:teuthology.orchestra.run.smithi040.stdout:Packager : CentOS BuildSystem 2023-09-29T23:22:29.559 INFO:teuthology.orchestra.run.smithi040.stdout:Vendor : CentOS 2023-09-29T23:22:29.559 INFO:teuthology.orchestra.run.smithi040.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-29T23:22:29.559 INFO:teuthology.orchestra.run.smithi040.stdout:Summary : Support tools for GRUB. 2023-09-29T23:22:29.559 INFO:teuthology.orchestra.run.smithi040.stdout:Description : 2023-09-29T23:22:29.559 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-29T23:22:29.560 INFO:teuthology.orchestra.run.smithi040.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-29T23:22:29.560 INFO:teuthology.orchestra.run.smithi040.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-29T23:22:29.560 INFO:teuthology.orchestra.run.smithi040.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-29T23:22:29.560 INFO:teuthology.orchestra.run.smithi040.stdout:hardware devices. 2023-09-29T23:22:29.560 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-29T23:22:29.560 INFO:teuthology.orchestra.run.smithi040.stdout:This subpackage provides tools for support of all platforms. 2023-09-29T23:22:29.561 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-29T23:22:29.561 INFO:teuthology.task.kernel:Updating grub on smithi040 to boot 4.18.0-514.el8.x86_64 2023-09-29T23:22:29.562 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-29T23:22:29.574 DEBUG:teuthology.orchestra.run.smithi135:> sudo /bin/ls /boot/loader/entries || true 2023-09-29T23:22:29.674 INFO:teuthology.orchestra.run.smithi135.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-29T23:22:29.674 INFO:teuthology.orchestra.run.smithi135.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-29T23:22:29.674 INFO:teuthology.orchestra.run.smithi135.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-29T23:22:29.674 INFO:teuthology.orchestra.run.smithi135.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-29T23:22:29.675 DEBUG:teuthology.orchestra.run.smithi135:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-29T23:22:29.798 DEBUG:teuthology.orchestra.run.smithi135:> sudo shutdown -r now 2023-09-29T23:22:31.002 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2023-09-29T23:22:32.727 INFO:teuthology.orchestra.run.smithi040.stderr:done 2023-09-29T23:22:32.729 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2023-09-29T23:22:32.798 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.yByy4pVDQL 2023-09-29T23:22:32.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yByy4pVDQL 2023-09-29T23:22:32.880 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.yByy4pVDQL 2023-09-29T23:22:33.024 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.yByy4pVDQL is 6KB 2023-09-29T23:22:33.072 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.yByy4pVDQL 2023-09-29T23:22:33.092 DEBUG:teuthology.orchestra.run.smithi040:> sudo /bin/ls /boot/loader/entries || true 2023-09-29T23:22:33.174 INFO:teuthology.orchestra.run.smithi040.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-29T23:22:33.174 INFO:teuthology.orchestra.run.smithi040.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-29T23:22:33.174 INFO:teuthology.orchestra.run.smithi040.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-29T23:22:33.174 INFO:teuthology.orchestra.run.smithi040.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-29T23:22:33.175 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-29T23:22:33.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2023-09-29T23:22:59.826 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-09-29T23:22:59.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-09-29T23:22:59.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:23:03.346 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-29T23:23:03.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-09-29T23:23:03.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:23:18.267 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2023-09-29T23:23:21.850 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-09-29T23:23:27.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-09-29T23:23:27.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:23:30.330 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2023-09-29T23:23:30.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-09-29T23:23:30.852 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:23:33.914 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-09-29T23:23:42.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-09-29T23:23:42.339 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:23:45.918 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-09-29T23:23:45.919 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:24:00.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2023-09-29T23:24:04.346 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2023-09-29T23:24:15.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-09-29T23:24:15.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:24:16.577 DEBUG:teuthology.orchestra.run.smithi135:> true 2023-09-29T23:24:18.034 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-09-29T23:24:18.034 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-514.el8.x86_64"... 2023-09-29T23:24:18.034 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2023-09-29T23:24:18.102 INFO:teuthology.orchestra.run.smithi135.stdout:4.18.0-514.el8.x86_64 2023-09-29T23:24:18.102 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-29T23:24:18.102 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-29T23:24:18.102 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-29T23:24:19.104 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-09-29T23:24:19.104 DEBUG:teuthology.orchestra.run.smithi135:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-29T23:24:19.192 INFO:teuthology.orchestra.run.smithi135.stdout:ttyS1 2023-09-29T23:24:19.206 DEBUG:teuthology.parallel:result is None 2023-09-29T23:24:19.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-09-29T23:24:19.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:24:19.634 DEBUG:teuthology.orchestra.run.smithi040:> true 2023-09-29T23:24:19.910 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-09-29T23:24:19.910 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-514.el8.x86_64"... 2023-09-29T23:24:19.910 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2023-09-29T23:24:19.930 INFO:teuthology.orchestra.run.smithi040.stdout:4.18.0-514.el8.x86_64 2023-09-29T23:24:19.931 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-29T23:24:19.931 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-29T23:24:19.931 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-29T23:24:20.932 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-29T23:24:20.932 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-29T23:24:21.004 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2023-09-29T23:24:21.022 DEBUG:teuthology.parallel:result is None 2023-09-29T23:24:21.022 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-29T23:24:21.031 INFO:teuthology.task.internal:Creating test directory... 2023-09-29T23:24:21.031 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-29T23:24:21.033 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-29T23:24:21.056 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-29T23:24:21.065 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-29T23:24:21.079 INFO:teuthology.task.internal:Creating archive directory... 2023-09-29T23:24:21.079 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-29T23:24:21.093 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-29T23:24:21.144 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-29T23:24:21.155 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-29T23:24:21.155 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-29T23:24:21.162 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-29T23:24:21.217 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T23:24:21.229 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T23:24:21.229 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T23:24:21.243 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T23:24:21.244 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-29T23:24:21.252 INFO:teuthology.task.internal:Configuring sudo... 2023-09-29T23:24:21.253 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-29T23:24:21.255 DEBUG:teuthology.orchestra.run.smithi135:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-29T23:24:21.301 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-29T23:24:21.310 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-29T23:24:21.311 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-29T23:24:21.325 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-29T23:24:21.359 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T23:24:21.427 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T23:24:21.479 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T23:24:21.563 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T23:24:21.606 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-29T23:24:21.606 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-29T23:24:21.678 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T23:24:21.726 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T23:24:21.777 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T23:24:21.861 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T23:24:21.906 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-29T23:24:21.906 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-29T23:24:21.974 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2023-09-29T23:24:21.976 DEBUG:teuthology.orchestra.run.smithi135:> sudo service rsyslog restart 2023-09-29T23:24:22.040 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-29T23:24:22.080 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-29T23:24:22.507 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-29T23:24:22.516 INFO:teuthology.task.internal:Starting timer... 2023-09-29T23:24:22.516 INFO:teuthology.run_tasks:Running task pcp... 2023-09-29T23:24:22.528 INFO:teuthology.run_tasks:Running task selinux... 2023-09-29T23:24:22.540 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-29T23:24:22.540 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2023-09-29T23:24:22.610 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2023-09-29T23:24:22.612 DEBUG:teuthology.orchestra.run.smithi135:> sudo service auditd rotate 2023-09-29T23:24:22.702 INFO:teuthology.orchestra.run.smithi135.stdout:Rotating logs: 2023-09-29T23:24:22.704 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-29T23:24:22.705 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2023-09-29T23:24:22.728 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2023-09-29T23:24:22.728 DEBUG:teuthology.orchestra.run.smithi135:> /usr/sbin/getenforce 2023-09-29T23:24:22.756 INFO:teuthology.orchestra.run.smithi135.stdout:Permissive 2023-09-29T23:24:22.756 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi135.front.sepia.ceph.com': 'permissive'} 2023-09-29T23:24:22.756 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-29T23:24:22.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:24:22.800 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-29T23:24:22.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:24:22.830 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-29T23:24:22.830 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2023-09-29T23:24:22.875 DEBUG:teuthology.orchestra.run.smithi135:> sudo /usr/sbin/setenforce permissive 2023-09-29T23:24:22.902 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-29T23:24:22.912 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-09-29T23:24:22.917 INFO:teuthology.repo_utils:Fetching main from origin 2023-09-29T23:24:23.038 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-29T23:24:23.054 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-29T23:24:23.055 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi040.front.sepia.ceph.com,smithi135.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-29T23:40:41.761 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi135.front.sepia.ceph.com')] 2023-09-29T23:40:41.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-09-29T23:40:41.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:40:41.837 DEBUG:teuthology.orchestra.run.smithi040:> true 2023-09-29T23:40:41.917 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2023-09-29T23:40:41.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-09-29T23:40:41.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T23:40:41.987 DEBUG:teuthology.orchestra.run.smithi135:> true 2023-09-29T23:40:42.057 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-09-29T23:40:42.058 INFO:teuthology.run_tasks:Running task clock... 2023-09-29T23:40:42.074 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-29T23:40:42.074 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-29T23:40:42.075 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T23:40:42.077 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-29T23:40:42.078 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T23:40:42.116 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-29T23:40:42.132 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-29T23:40:42.132 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-29T23:40:42.146 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-29T23:40:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2023-09-29T23:40:42.212 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: ntpd: command not found 2023-09-29T23:40:42.219 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2023-09-29T23:40:42.223 INFO:teuthology.orchestra.run.smithi135.stdout:506 Cannot talk to daemon 2023-09-29T23:40:42.236 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-29T23:40:42.237 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-29T23:40:42.250 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-29T23:40:42.252 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-29T23:40:42.325 INFO:teuthology.orchestra.run.smithi135.stderr:bash: ntpq: command not found 2023-09-29T23:40:42.328 INFO:teuthology.orchestra.run.smithi040.stderr:bash: ntpq: command not found 2023-09-29T23:40:42.329 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-29T23:40:42.329 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2023-09-29T23:40:42.329 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:40:42.330 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:40:42.330 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:40:42.330 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:40:42.333 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-29T23:40:42.333 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2023-09-29T23:40:42.333 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:40:42.333 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:40:42.333 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:40:42.333 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-29T23:40:42.334 INFO:teuthology.run_tasks:Running task install... 2023-09-29T23:40:42.343 DEBUG:teuthology.task.install:project ceph 2023-09-29T23:40:42.343 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-29T23:40:42.343 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-29T23:40:42.343 INFO:teuthology.task.install:Using flavor: default 2023-09-29T23:40:42.348 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-09-29T23:40:42.349 INFO:teuthology.task.install:extra packages: [] 2023-09-29T23:40:42.349 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-29T23:40:42.351 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T23:40:42.352 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-29T23:40:42.353 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-29T23:40:42.573 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-29T23:40:42.573 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-29T23:40:42.579 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-29T23:40:42.579 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-29T23:40:42.687 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-29T23:40:42.687 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-29T23:40:42.687 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-29T23:40:42.715 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-09-29T23:40:42.716 DEBUG:teuthology.orchestra.run.smithi135:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-29T23:40:42.739 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-29T23:40:42.740 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-29T23:40:42.740 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-29T23:40:42.776 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-09-29T23:40:42.776 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-29T23:40:42.784 DEBUG:teuthology.orchestra.run.smithi135:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-29T23:40:42.853 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-29T23:40:42.861 DEBUG:teuthology.orchestra.run.smithi135:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-29T23:40:42.927 INFO:teuthology.orchestra.run.smithi135.stdout:check_obsoletes = 1 2023-09-29T23:40:42.928 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum clean all 2023-09-29T23:40:42.940 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-29T23:40:42.972 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2023-09-29T23:40:42.974 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2023-09-29T23:40:43.204 INFO:teuthology.orchestra.run.smithi135.stdout:76 files removed 2023-09-29T23:40:43.232 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-radosgw 2023-09-29T23:40:43.287 INFO:teuthology.orchestra.run.smithi040.stdout:76 files removed 2023-09-29T23:40:43.320 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw 2023-09-29T23:40:44.058 INFO:teuthology.orchestra.run.smithi135.stdout:ceph packages for x86_64 186 kB/s | 78 kB 00:00 2023-09-29T23:40:44.233 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 175 kB/s | 78 kB 00:00 2023-09-29T23:40:44.418 INFO:teuthology.orchestra.run.smithi135.stdout:ceph noarch packages 29 kB/s | 9.2 kB 00:00 2023-09-29T23:40:44.551 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 34 kB/s | 9.2 kB 00:00 2023-09-29T23:40:44.677 INFO:teuthology.orchestra.run.smithi135.stdout:ceph source packages 6.3 kB/s | 1.5 kB 00:00 2023-09-29T23:40:44.884 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 4.8 kB/s | 1.5 kB 00:00 2023-09-29T23:54:53.539 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - AppStream 40 kB/s | 33 MB 14:08 2023-09-29T23:54:54.414 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - AppStream 40 kB/s | 33 MB 14:09 2023-09-29T23:55:06.146 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - BaseOS 8.1 MB/s | 48 MB 00:05 2023-09-29T23:55:06.439 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - BaseOS 7.9 MB/s | 48 MB 00:06 2023-09-29T23:55:13.513 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - Extras 75 kB/s | 18 kB 00:00 2023-09-29T23:55:14.121 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - Extras 47 kB/s | 18 kB 00:00 2023-09-29T23:55:14.139 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - Extras common packages 15 kB/s | 6.9 kB 00:00 2023-09-29T23:55:14.749 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 6.9 kB 00:00 2023-09-29T23:55:14.940 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - PowerTools 10 MB/s | 6.5 MB 00:00 2023-09-29T23:55:15.465 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 8 - PowerTools 12 MB/s | 6.5 MB 00:00 2023-09-29T23:55:16.400 INFO:teuthology.orchestra.run.smithi135.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-09-29T23:55:17.000 INFO:teuthology.orchestra.run.smithi040.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-09-29T23:55:17.826 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 24 MB/s | 16 MB 00:00 2023-09-29T23:55:17.953 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-09-29T23:55:22.109 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 65 kB/s | 24 kB 00:00 2023-09-29T23:55:22.126 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 337 kB/s | 24 kB 00:00 2023-09-29T23:55:26.504 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-29T23:55:26.506 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-29T23:55:26.506 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-09-29T23:55:26.506 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-29T23:55:26.507 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-29T23:55:26.507 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-29T23:55:26.507 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrading: 2023-09-29T23:55:26.507 INFO:teuthology.orchestra.run.smithi135.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-29T23:55:26.507 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-29T23:55:26.507 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-09-29T23:55:26.507 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-29T23:55:26.507 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-29T23:55:26.507 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-29T23:55:26.507 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-29T23:55:26.507 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-29T23:55:26.508 INFO:teuthology.orchestra.run.smithi135.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-29T23:55:26.508 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-29T23:55:26.508 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-29T23:55:26.508 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-29T23:55:26.508 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-29T23:55:26.508 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-29T23:55:26.508 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-29T23:55:26.508 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-29T23:55:26.508 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-29T23:55:26.508 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-29T23:55:26.508 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-29T23:55:26.508 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-29T23:55:26.509 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-29T23:55:26.509 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-29T23:55:26.509 INFO:teuthology.orchestra.run.smithi135.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-29T23:55:26.509 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-29T23:55:26.509 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-29T23:55:26.509 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-29T23:55:26.509 INFO:teuthology.orchestra.run.smithi135.stdout:Install 21 Packages 2023-09-29T23:55:26.509 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrade 2 Packages 2023-09-29T23:55:26.509 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-29T23:55:26.510 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 63 M 2023-09-29T23:55:26.510 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-29T23:55:27.023 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-29T23:55:27.026 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-29T23:55:27.026 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-09-29T23:55:27.027 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-29T23:55:27.027 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-29T23:55:27.027 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-29T23:55:27.027 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2023-09-29T23:55:27.027 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-29T23:55:27.027 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-29T23:55:27.027 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-09-29T23:55:27.027 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-29T23:55:27.027 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-29T23:55:27.027 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-29T23:55:27.027 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-29T23:55:27.028 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-29T23:55:27.028 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-29T23:55:27.028 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-29T23:55:27.028 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-29T23:55:27.028 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-29T23:55:27.028 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-29T23:55:27.028 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-29T23:55:27.028 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-29T23:55:27.028 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-29T23:55:27.028 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-29T23:55:27.028 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-29T23:55:27.028 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-29T23:55:27.029 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-29T23:55:27.029 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-29T23:55:27.029 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-29T23:55:27.029 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-29T23:55:27.029 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-29T23:55:27.029 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-29T23:55:27.029 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-29T23:55:27.029 INFO:teuthology.orchestra.run.smithi040.stdout:Install 21 Packages 2023-09-29T23:55:27.029 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 2 Packages 2023-09-29T23:55:27.029 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-29T23:55:27.030 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 63 M 2023-09-29T23:55:27.030 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-29T23:55:27.936 INFO:teuthology.orchestra.run.smithi135.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 5.5 MB/s | 5.2 MB 00:00 2023-09-29T23:55:28.036 INFO:teuthology.orchestra.run.smithi135.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 244 kB/s | 24 kB 00:00 2023-09-29T23:55:28.278 INFO:teuthology.orchestra.run.smithi135.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 3.2 MB/s | 796 kB 00:00 2023-09-29T23:55:28.363 INFO:teuthology.orchestra.run.smithi040.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 4.3 MB/s | 5.2 MB 00:01 2023-09-29T23:55:28.403 INFO:teuthology.orchestra.run.smithi135.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 4.0 MB/s | 506 kB 00:00 2023-09-29T23:55:28.452 INFO:teuthology.orchestra.run.smithi040.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 275 kB/s | 24 kB 00:00 2023-09-29T23:55:28.653 INFO:teuthology.orchestra.run.smithi040.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 3.9 MB/s | 796 kB 00:00 2023-09-29T23:55:28.811 INFO:teuthology.orchestra.run.smithi040.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 3.1 MB/s | 506 kB 00:00 2023-09-29T23:55:29.504 INFO:teuthology.orchestra.run.smithi135.stdout:(5/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8. 6.7 MB/s | 17 MB 00:02 2023-09-29T23:55:29.637 INFO:teuthology.orchestra.run.smithi040.stdout:(5/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 6.7 MB/s | 5.6 MB 00:00 2023-09-29T23:55:29.637 INFO:teuthology.orchestra.run.smithi135.stdout:(6/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 4.5 MB/s | 5.6 MB 00:01 2023-09-29T23:55:29.654 INFO:teuthology.orchestra.run.smithi135.stdout:(7/23): python3-ceph-argparse-17.2.6-1165.g93dc 297 kB/s | 45 kB 00:00 2023-09-29T23:55:29.703 INFO:teuthology.orchestra.run.smithi040.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 672 kB/s | 45 kB 00:00 2023-09-29T23:55:29.704 INFO:teuthology.orchestra.run.smithi135.stdout:(8/23): python3-ceph-common-17.2.6-1165.g93dce5 1.7 MB/s | 115 kB 00:00 2023-09-29T23:55:29.738 INFO:teuthology.orchestra.run.smithi135.stdout:(9/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 2.2 MB/s | 190 kB 00:00 2023-09-29T23:55:29.770 INFO:teuthology.orchestra.run.smithi040.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 1.7 MB/s | 115 kB 00:00 2023-09-29T23:55:29.788 INFO:teuthology.orchestra.run.smithi135.stdout:(10/23): python3-rados-17.2.6-1165.g93dce5e1.el 4.3 MB/s | 370 kB 00:00 2023-09-29T23:55:29.813 INFO:teuthology.orchestra.run.smithi135.stdout:(11/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 4.7 MB/s | 358 kB 00:00 2023-09-29T23:55:29.846 INFO:teuthology.orchestra.run.smithi040.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 2.5 MB/s | 190 kB 00:00 2023-09-29T23:55:29.855 INFO:teuthology.orchestra.run.smithi135.stdout:(12/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 1.6 MB/s | 107 kB 00:00 2023-09-29T23:55:29.913 INFO:teuthology.orchestra.run.smithi040.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 5.4 MB/s | 370 kB 00:00 2023-09-29T23:55:30.130 INFO:teuthology.orchestra.run.smithi040.stdout:(10/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8 5.6 MB/s | 17 MB 00:02 2023-09-29T23:55:30.155 INFO:teuthology.orchestra.run.smithi040.stdout:(11/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 1.4 MB/s | 358 kB 00:00 2023-09-29T23:55:30.180 INFO:teuthology.orchestra.run.smithi040.stdout:(12/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 2.1 MB/s | 107 kB 00:00 2023-09-29T23:55:31.522 INFO:teuthology.orchestra.run.smithi135.stdout:(13/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 5.2 MB/s | 23 MB 00:04 2023-09-29T23:55:32.789 INFO:teuthology.orchestra.run.smithi040.stdout:(13/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 4.2 MB/s | 23 MB 00:05 2023-09-29T23:59:37.857 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] librdkafka-0.11.4-3.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/librdkafka-0.11.4-3.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-09-29T23:59:37.857 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] lttng-ust-2.8.1-11.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/lttng-ust-2.8.1-11.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-09-29T23:59:37.975 INFO:teuthology.orchestra.run.smithi040.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 kB/s | 259 kB 04:07 2023-09-29T23:59:38.026 INFO:teuthology.orchestra.run.smithi040.stdout:(15/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.4 kB/s | 354 kB 04:07 2023-09-29T23:59:38.184 INFO:teuthology.orchestra.run.smithi040.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 225 kB/s | 47 kB 00:00 2023-09-29T23:59:38.535 INFO:teuthology.orchestra.run.smithi040.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 168 kB/s | 59 kB 00:00 2023-09-29T23:59:38.610 INFO:teuthology.orchestra.run.smithi040.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 525 kB/s | 306 kB 00:00 2023-09-29T23:59:38.702 INFO:teuthology.orchestra.run.smithi040.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 450 kB/s | 75 kB 00:00 2023-09-29T23:59:38.917 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] librdkafka-0.11.4-3.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/librdkafka-0.11.4-3.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-09-29T23:59:38.986 INFO:teuthology.orchestra.run.smithi040.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 4.6 MB/s | 1.7 MB 00:00 2023-09-29T23:59:39.015 INFO:teuthology.orchestra.run.smithi135.stdout:(14/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.4 kB/s | 354 kB 04:09 2023-09-29T23:59:39.099 INFO:teuthology.orchestra.run.smithi135.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 563 kB/s | 47 kB 00:00 2023-09-29T23:59:39.583 INFO:teuthology.orchestra.run.smithi135.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 634 kB/s | 306 kB 00:00 2023-09-29T23:59:39.674 INFO:teuthology.orchestra.run.smithi135.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 642 kB/s | 59 kB 00:00 2023-09-29T23:59:39.703 INFO:teuthology.orchestra.run.smithi040.stdout:(21/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 3.5 MB/s | 3.5 MB 00:01 2023-09-29T23:59:39.775 INFO:teuthology.orchestra.run.smithi135.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 750 kB/s | 75 kB 00:00 2023-09-29T23:59:40.011 INFO:teuthology.orchestra.run.smithi040.stdout:(22/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 3.5 MB/s | 3.6 MB 00:01 2023-09-29T23:59:40.050 INFO:teuthology.orchestra.run.smithi135.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 6.3 MB/s | 1.7 MB 00:00 2023-09-29T23:59:40.983 INFO:teuthology.orchestra.run.smithi135.stdout:(20/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 3.7 MB/s | 3.5 MB 00:00 2023-09-29T23:59:41.442 INFO:teuthology.orchestra.run.smithi135.stdout:(21/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 7.8 MB/s | 3.6 MB 00:00 2023-09-30T00:00:31.000 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] lttng-ust-2.8.1-11.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/lttng-ust-2.8.1-11.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-09-30T00:00:31.311 INFO:teuthology.orchestra.run.smithi135.stdout:(22/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 881 B/s | 259 kB 05:01 2023-09-30T00:00:32.523 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] python3-prettytable-0.7.2-14.el8.noarch.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-prettytable-0.7.2-14.el8.noarch.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-09-30T00:00:32.695 INFO:teuthology.orchestra.run.smithi135.stdout:(23/23): python3-prettytable-0.7.2-14.el8.noarc 149 B/s | 44 kB 05:01 2023-09-30T00:00:32.696 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:00:32.696 INFO:teuthology.orchestra.run.smithi135.stdout:Total 212 kB/s | 63 MB 05:06 2023-09-30T00:00:32.737 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:00:32.783 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:00:32.783 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:00:33.149 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:00:33.152 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:00:33.791 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] python3-prettytable-0.7.2-14.el8.noarch.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-prettytable-0.7.2-14.el8.noarch.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-09-30T00:00:34.005 INFO:teuthology.orchestra.run.smithi040.stdout:(23/23): python3-prettytable-0.7.2-14.el8.noarc 149 B/s | 44 kB 05:01 2023-09-30T00:00:34.008 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:00:34.008 INFO:teuthology.orchestra.run.smithi040.stdout:Total 211 kB/s | 63 MB 05:06 2023-09-30T00:00:34.052 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:00:34.104 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:00:34.104 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:00:34.361 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:00:34.515 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:00:34.519 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:00:34.570 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-30T00:00:34.783 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:00:35.161 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:00:35.324 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:00:35.403 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:00:35.712 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:00:35.719 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:00:35.941 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:00:35.950 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-30T00:00:36.116 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:00:36.325 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:00:36.507 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:00:36.575 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:00:36.725 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:00:36.766 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:00:36.912 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-30T00:00:36.918 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:00:37.139 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-30T00:00:37.183 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:00:37.299 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:00:37.366 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:00:37.806 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:00:37.854 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:00:38.010 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:00:38.015 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:00:38.081 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:00:38.227 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:00:38.292 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T00:00:38.422 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-30T00:00:38.470 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:00:38.651 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-30T00:00:38.709 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:00:38.866 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:00:38.977 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T00:00:39.107 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:00:39.185 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:00:39.441 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T00:00:39.484 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:00:39.624 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-30T00:00:39.649 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:00:39.718 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:00:39.844 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T00:00:39.912 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T00:00:40.041 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T00:00:40.088 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:00:40.330 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:00:40.583 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T00:00:40.715 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:00:40.800 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:00:41.054 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T00:00:41.417 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-30T00:00:41.599 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T00:00:41.774 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T00:00:42.690 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:00:42.882 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:00:43.474 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:00:43.702 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:00:43.811 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:00:43.973 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:00:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:00:45.140 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:00:45.818 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:00:46.123 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:00:46.225 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:00:46.448 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:01:02.726 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:01:02.726 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /sys 2023-09-30T00:01:02.726 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /proc 2023-09-30T00:01:02.727 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /dev 2023-09-30T00:01:02.727 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /run 2023-09-30T00:01:02.727 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /mnt 2023-09-30T00:01:02.727 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /var/tmp 2023-09-30T00:01:02.727 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /home 2023-09-30T00:01:02.727 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /tmp 2023-09-30T00:01:02.727 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /dev 2023-09-30T00:01:02.727 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:01:04.433 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:01:04.511 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:01:04.512 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:01:04.714 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:01:04.714 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:01:05.431 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:01:05.431 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T00:01:05.431 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:01:05.431 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:01:05.431 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:01:05.431 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:01:05.431 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T00:01:05.432 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T00:01:05.432 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T00:01:05.432 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T00:01:05.432 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:01:05.432 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T00:01:05.432 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:01:05.432 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:01:05.432 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-30T00:01:05.432 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-30T00:01:05.432 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T00:01:05.432 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-30T00:01:05.432 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T00:01:05.433 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-30T00:01:05.433 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-30T00:01:05.433 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-30T00:01:05.433 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:01:05.433 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T00:01:05.433 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T00:01:06.690 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:01:06.691 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:01:06.691 INFO:teuthology.orchestra.run.smithi135.stdout:Upgraded: 2023-09-30T00:01:06.691 INFO:teuthology.orchestra.run.smithi135.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.691 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.691 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:01:06.691 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.691 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.691 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.691 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.691 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T00:01:06.692 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.692 INFO:teuthology.orchestra.run.smithi135.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T00:01:06.692 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-30T00:01:06.692 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.692 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-30T00:01:06.692 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.692 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T00:01:06.692 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T00:01:06.692 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.693 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.693 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.693 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T00:01:06.693 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.693 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.693 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:06.693 INFO:teuthology.orchestra.run.smithi135.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T00:01:06.693 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:01:06.693 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:01:06.969 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-test 2023-09-30T00:01:07.668 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:01:07.668 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2023-09-30T00:01:07.669 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2023-09-30T00:01:07.669 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /dev 2023-09-30T00:01:07.669 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /run 2023-09-30T00:01:07.669 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2023-09-30T00:01:07.669 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2023-09-30T00:01:07.669 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2023-09-30T00:01:07.669 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2023-09-30T00:01:07.669 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /dev 2023-09-30T00:01:07.669 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:01:07.820 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:05:45 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:01:09.423 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:01:09.424 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:01:09.424 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-09-30T00:01:09.424 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:01:09.424 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:01:09.424 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T00:01:09.424 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-09-30T00:01:09.424 INFO:teuthology.orchestra.run.smithi135.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-30T00:01:09.424 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T00:01:09.424 INFO:teuthology.orchestra.run.smithi135.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-30T00:01:09.424 INFO:teuthology.orchestra.run.smithi135.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-30T00:01:09.425 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-30T00:01:09.425 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:01:09.425 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:01:09.425 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:01:09.425 INFO:teuthology.orchestra.run.smithi135.stdout:Install 6 Packages 2023-09-30T00:01:09.425 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:01:09.425 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 53 M 2023-09-30T00:01:09.426 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 233 M 2023-09-30T00:01:09.426 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:01:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:01:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:01:09.625 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:01:09.770 INFO:teuthology.orchestra.run.smithi135.stdout:(1/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 581 kB/s | 175 kB 00:00 2023-09-30T00:01:09.840 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:01:09.840 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:01:10.703 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:01:10.704 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T00:01:10.704 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:01:10.704 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:01:10.704 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:01:10.704 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:01:10.704 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T00:01:10.704 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T00:01:10.704 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T00:01:10.704 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T00:01:10.704 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:01:10.705 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T00:01:10.705 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:01:10.705 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:01:10.705 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-30T00:01:10.705 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-30T00:01:10.705 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T00:01:10.705 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-30T00:01:10.705 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T00:01:10.705 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-30T00:01:10.705 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-30T00:01:10.705 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-30T00:01:10.705 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:01:10.705 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T00:01:10.706 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T00:01:11.946 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:01:11.947 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:01:11.947 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2023-09-30T00:01:11.947 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.947 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.947 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:01:11.947 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.947 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.948 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.948 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.948 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T00:01:11.948 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.948 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T00:01:11.948 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-30T00:01:11.948 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.948 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-30T00:01:11.948 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.948 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T00:01:11.948 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T00:01:11.948 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.949 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.949 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.949 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T00:01:11.949 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.949 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.949 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:01:11.949 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T00:01:11.949 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:01:11.949 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:01:12.120 INFO:teuthology.orchestra.run.smithi135.stdout:(2/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 19 MB/s | 52 MB 00:02 2023-09-30T00:01:12.164 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-test 2023-09-30T00:01:13.148 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:05:51 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:01:14.897 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:01:14.898 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:01:14.898 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-09-30T00:01:14.899 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:01:14.899 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:01:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T00:01:14.899 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-09-30T00:01:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-30T00:01:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T00:01:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-30T00:01:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-30T00:01:14.899 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-30T00:01:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:01:14.900 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:01:14.900 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:01:14.900 INFO:teuthology.orchestra.run.smithi040.stdout:Install 6 Packages 2023-09-30T00:01:14.900 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:01:14.900 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 53 M 2023-09-30T00:01:14.901 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 233 M 2023-09-30T00:01:14.901 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:01:15.302 INFO:teuthology.orchestra.run.smithi040.stdout:(1/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 811 kB/s | 175 kB 00:00 2023-09-30T00:01:18.069 INFO:teuthology.orchestra.run.smithi040.stdout:(2/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 17 MB/s | 52 MB 00:02 2023-09-30T00:06:09.773 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] jq-1.6-7.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/jq-1.6-7.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-09-30T00:06:09.823 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] oniguruma-6.8.2-2.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/oniguruma-6.8.2-2.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-09-30T00:06:10.038 INFO:teuthology.orchestra.run.smithi135.stdout:(3/6): jq-1.6-7.el8.x86_64.rpm 690 B/s | 202 kB 05:00 2023-09-30T00:06:10.121 INFO:teuthology.orchestra.run.smithi135.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 639 B/s | 187 kB 05:00 2023-09-30T00:06:10.146 INFO:teuthology.orchestra.run.smithi135.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 651 kB/s | 71 kB 00:00 2023-09-30T00:06:13.121 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] socat-1.7.4.1-1.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-1.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-09-30T00:06:13.388 INFO:teuthology.orchestra.run.smithi135.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.1 kB/s | 323 kB 05:01 2023-09-30T00:06:13.389 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:06:13.389 INFO:teuthology.orchestra.run.smithi135.stdout:Total 177 kB/s | 53 MB 05:03 2023-09-30T00:06:13.428 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:06:13.449 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:06:13.449 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:06:13.734 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:06:13.737 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:06:14.274 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:06:14.506 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-30T00:06:14.743 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-30T00:06:14.874 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T00:06:14.934 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T00:06:15.156 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-30T00:06:15.303 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] jq-1.6-7.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/jq-1.6-7.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-09-30T00:06:15.315 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:06:15.353 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] oniguruma-6.8.2-2.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/oniguruma-6.8.2-2.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-09-30T00:06:15.606 INFO:teuthology.orchestra.run.smithi040.stdout:(3/6): jq-1.6-7.el8.x86_64.rpm 690 B/s | 202 kB 05:00 2023-09-30T00:06:15.631 INFO:teuthology.orchestra.run.smithi040.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 639 B/s | 187 kB 05:00 2023-09-30T00:06:15.665 INFO:teuthology.orchestra.run.smithi040.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-09-30T00:06:19.071 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] socat-1.7.4.1-1.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-1.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-09-30T00:06:19.323 INFO:teuthology.orchestra.run.smithi040.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.1 kB/s | 323 kB 05:01 2023-09-30T00:06:19.323 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:06:19.324 INFO:teuthology.orchestra.run.smithi040.stdout:Total 177 kB/s | 53 MB 05:04 2023-09-30T00:06:19.365 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:06:19.389 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:06:19.389 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:06:19.704 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:06:19.708 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:06:20.219 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:06:20.349 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:06:20.446 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-30T00:06:20.604 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:06:20.649 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-30T00:06:20.776 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T00:06:20.864 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T00:06:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:06:20.933 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T00:06:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T00:06:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-30T00:06:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-30T00:06:20.934 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-30T00:06:21.036 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-30T00:06:21.185 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:06:21.575 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-30T00:06:21.575 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:06:21.575 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:06:21.575 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:06:21.576 INFO:teuthology.orchestra.run.smithi135.stdout: jq-1.6-7.el8.x86_64 2023-09-30T00:06:21.576 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:06:21.576 INFO:teuthology.orchestra.run.smithi135.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T00:06:21.576 INFO:teuthology.orchestra.run.smithi135.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-30T00:06:21.576 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T00:06:21.576 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:06:21.576 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:06:21.771 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph 2023-09-30T00:06:22.626 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:11:00 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:06:24.219 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:06:24.222 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:06:24.223 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-09-30T00:06:24.223 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:06:24.223 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:06:24.223 INFO:teuthology.orchestra.run.smithi135.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T00:06:24.223 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-09-30T00:06:24.223 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T00:06:24.223 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T00:06:24.223 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T00:06:24.223 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T00:06:24.224 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:06:24.224 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T00:06:24.224 INFO:teuthology.orchestra.run.smithi135.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T00:06:24.224 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-30T00:06:24.224 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T00:06:24.224 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T00:06:24.224 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-30T00:06:24.224 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T00:06:24.224 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T00:06:24.224 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T00:06:24.225 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-30T00:06:24.225 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-30T00:06:24.225 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T00:06:24.225 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T00:06:24.225 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T00:06:24.225 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-30T00:06:24.225 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-30T00:06:24.225 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-30T00:06:24.225 INFO:teuthology.orchestra.run.smithi135.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T00:06:24.226 INFO:teuthology.orchestra.run.smithi135.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T00:06:24.226 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T00:06:24.226 INFO:teuthology.orchestra.run.smithi135.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T00:06:24.226 INFO:teuthology.orchestra.run.smithi135.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T00:06:24.226 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T00:06:24.226 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T00:06:24.226 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-30T00:06:24.226 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T00:06:24.226 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:06:24.226 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:06:24.226 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:06:24.227 INFO:teuthology.orchestra.run.smithi135.stdout:Install 32 Packages 2023-09-30T00:06:24.227 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:06:24.227 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 28 M 2023-09-30T00:06:24.227 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 110 M 2023-09-30T00:06:24.228 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:06:25.058 INFO:teuthology.orchestra.run.smithi135.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 56 kB/s | 6.4 kB 00:00 2023-09-30T00:06:25.383 INFO:teuthology.orchestra.run.smithi135.stdout:(2/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 12 MB/s | 3.9 MB 00:00 2023-09-30T00:06:25.658 INFO:teuthology.orchestra.run.smithi135.stdout:(3/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 3.2 MB/s | 2.3 MB 00:00 2023-09-30T00:06:25.767 INFO:teuthology.orchestra.run.smithi135.stdout:(4/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 2.1 MB/s | 238 kB 00:00 2023-09-30T00:06:25.809 INFO:teuthology.orchestra.run.smithi135.stdout:(5/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 1.8 MB/s | 1.6 MB 00:00 2023-09-30T00:06:26.226 INFO:teuthology.orchestra.run.smithi135.stdout:(6/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86_ 20 MB/s | 17 MB 00:00 2023-09-30T00:06:26.669 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:06:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:06:27.180 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:06:27.180 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T00:06:27.181 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T00:06:27.181 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-30T00:06:27.181 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-30T00:06:27.181 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-30T00:06:27.942 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-30T00:06:27.942 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:06:27.942 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:06:27.942 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:06:27.942 INFO:teuthology.orchestra.run.smithi040.stdout: jq-1.6-7.el8.x86_64 2023-09-30T00:06:27.942 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:06:27.942 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T00:06:27.943 INFO:teuthology.orchestra.run.smithi040.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-30T00:06:27.943 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T00:06:27.943 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:06:27.943 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:06:28.185 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph 2023-09-30T00:06:29.164 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:11:07 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:06:30.927 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:06:30.930 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:06:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-09-30T00:06:30.931 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:06:30.931 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:06:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T00:06:30.931 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-09-30T00:06:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T00:06:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T00:06:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T00:06:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T00:06:30.931 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:06:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T00:06:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T00:06:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-30T00:06:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T00:06:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T00:06:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-30T00:06:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T00:06:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T00:06:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T00:06:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-30T00:06:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-30T00:06:30.932 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T00:06:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T00:06:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T00:06:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-30T00:06:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-30T00:06:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-30T00:06:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T00:06:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T00:06:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T00:06:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T00:06:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T00:06:30.933 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T00:06:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T00:06:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-30T00:06:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T00:06:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:06:30.934 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:06:30.934 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:06:30.934 INFO:teuthology.orchestra.run.smithi040.stdout:Install 32 Packages 2023-09-30T00:06:30.934 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:06:30.935 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 28 M 2023-09-30T00:06:30.935 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 110 M 2023-09-30T00:06:30.935 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:06:31.391 INFO:teuthology.orchestra.run.smithi040.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 35 kB/s | 6.4 kB 00:00 2023-09-30T00:06:31.566 INFO:teuthology.orchestra.run.smithi040.stdout:(2/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 6.4 MB/s | 2.3 MB 00:00 2023-09-30T00:06:31.599 INFO:teuthology.orchestra.run.smithi040.stdout:(3/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 4.0 MB/s | 1.6 MB 00:00 2023-09-30T00:06:31.700 INFO:teuthology.orchestra.run.smithi040.stdout:(4/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 2.3 MB/s | 238 kB 00:00 2023-09-30T00:06:32.425 INFO:teuthology.orchestra.run.smithi040.stdout:(5/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86_ 19 MB/s | 17 MB 00:00 2023-09-30T00:06:32.492 INFO:teuthology.orchestra.run.smithi040.stdout:(6/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 3.6 MB/s | 3.9 MB 00:01 2023-09-30T00:10:34.255 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] python3-mako-1.0.6-14.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-mako-1.0.6-14.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:10:34.255 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] python3-markupsafe-0.23-19.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-markupsafe-0.23-19.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:10:34.255 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] python3-pyOpenSSL-19.0.0-1.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pyOpenSSL-19.0.0-1.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:10:34.392 INFO:teuthology.orchestra.run.smithi135.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 160 B/s | 39 kB 04:08 2023-09-30T00:10:34.417 INFO:teuthology.orchestra.run.smithi135.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 427 B/s | 103 kB 04:08 2023-09-30T00:10:34.443 INFO:teuthology.orchestra.run.smithi135.stdout:(9/32): python3-mako-1.0.6-14.el8.noarch.rpm 646 B/s | 157 kB 04:08 2023-09-30T00:10:34.477 INFO:teuthology.orchestra.run.smithi135.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 677 kB/s | 57 kB 00:00 2023-09-30T00:10:34.502 INFO:teuthology.orchestra.run.smithi135.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 5.3 MB/s | 457 kB 00:00 2023-09-30T00:10:34.610 INFO:teuthology.orchestra.run.smithi135.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.0 MB/s | 109 kB 00:00 2023-09-30T00:10:34.636 INFO:teuthology.orchestra.run.smithi135.stdout:(13/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.3 MB/s | 249 kB 00:00 2023-09-30T00:10:34.678 INFO:teuthology.orchestra.run.smithi135.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 3.1 MB/s | 641 kB 00:00 2023-09-30T00:10:34.762 INFO:teuthology.orchestra.run.smithi135.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 290 kB/s | 44 kB 00:00 2023-09-30T00:10:34.787 INFO:teuthology.orchestra.run.smithi135.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.2 MB/s | 185 kB 00:00 2023-09-30T00:10:34.812 INFO:teuthology.orchestra.run.smithi135.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2023-09-30T00:10:34.829 INFO:teuthology.orchestra.run.smithi135.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 265 kB/s | 11 kB 00:00 2023-09-30T00:10:34.846 INFO:teuthology.orchestra.run.smithi135.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 525 kB/s | 18 kB 00:00 2023-09-30T00:10:34.880 INFO:teuthology.orchestra.run.smithi135.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.2 MB/s | 384 kB 00:00 2023-09-30T00:10:34.905 INFO:teuthology.orchestra.run.smithi135.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 646 kB/s | 49 kB 00:00 2023-09-30T00:10:34.930 INFO:teuthology.orchestra.run.smithi135.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 700 kB/s | 59 kB 00:00 2023-09-30T00:10:34.956 INFO:teuthology.orchestra.run.smithi135.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 309 kB/s | 16 kB 00:00 2023-09-30T00:10:34.981 INFO:teuthology.orchestra.run.smithi135.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 372 kB/s | 19 kB 00:00 2023-09-30T00:10:35.006 INFO:teuthology.orchestra.run.smithi135.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 477 kB/s | 24 kB 00:00 2023-09-30T00:10:35.031 INFO:teuthology.orchestra.run.smithi135.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 583 kB/s | 29 kB 00:00 2023-09-30T00:10:35.056 INFO:teuthology.orchestra.run.smithi135.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 549 kB/s | 27 kB 00:00 2023-09-30T00:10:35.090 INFO:teuthology.orchestra.run.smithi135.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2023-09-30T00:10:35.116 INFO:teuthology.orchestra.run.smithi135.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.0 MB/s | 255 kB 00:00 2023-09-30T00:10:35.141 INFO:teuthology.orchestra.run.smithi135.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.9 MB/s | 251 kB 00:00 2023-09-30T00:10:35.166 INFO:teuthology.orchestra.run.smithi135.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2023-09-30T00:10:35.191 INFO:teuthology.orchestra.run.smithi135.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2023-09-30T00:10:35.192 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:10:35.192 INFO:teuthology.orchestra.run.smithi135.stdout:Total 116 kB/s | 28 MB 04:10 2023-09-30T00:10:35.271 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:10:35.308 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:10:35.308 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:10:35.521 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:10:35.524 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:10:36.580 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:10:36.773 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-30T00:10:36.967 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-30T00:10:37.184 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-30T00:10:37.295 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] python3-mako-1.0.6-14.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-mako-1.0.6-14.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:10:37.377 INFO:teuthology.orchestra.run.smithi040.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 654 B/s | 157 kB 04:05 2023-09-30T00:10:37.399 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-30T00:10:37.580 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-30T00:10:37.776 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-30T00:10:37.929 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-30T00:10:38.131 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-30T00:10:38.338 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-30T00:10:38.542 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-30T00:10:38.778 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-30T00:10:38.986 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-30T00:10:39.350 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-30T00:10:39.595 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-30T00:10:39.761 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-30T00:10:39.982 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-30T00:10:40.193 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T00:10:40.475 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-30T00:10:40.664 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-30T00:10:40.861 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-30T00:10:41.022 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-30T00:10:41.256 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-30T00:10:41.471 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-30T00:10:41.582 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] python3-markupsafe-0.23-19.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-markupsafe-0.23-19.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:10:41.582 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] python3-pyOpenSSL-19.0.0-1.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pyOpenSSL-19.0.0-1.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:10:41.619 INFO:teuthology.orchestra.run.smithi040.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 159 B/s | 39 kB 04:09 2023-09-30T00:10:41.669 INFO:teuthology.orchestra.run.smithi040.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 9.0 MB/s | 457 kB 00:00 2023-09-30T00:10:41.689 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-30T00:10:41.694 INFO:teuthology.orchestra.run.smithi040.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 425 B/s | 103 kB 04:09 2023-09-30T00:10:41.803 INFO:teuthology.orchestra.run.smithi040.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.8 MB/s | 249 kB 00:00 2023-09-30T00:10:41.828 INFO:teuthology.orchestra.run.smithi040.stdout:(12/32): python3-cryptography-3.2.1-6.el8.x86_6 4.7 MB/s | 641 kB 00:00 2023-09-30T00:10:41.853 INFO:teuthology.orchestra.run.smithi040.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 2.1 MB/s | 109 kB 00:00 2023-09-30T00:10:41.954 INFO:teuthology.orchestra.run.smithi040.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 348 kB/s | 44 kB 00:00 2023-09-30T00:10:41.979 INFO:teuthology.orchestra.run.smithi040.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.4 MB/s | 185 kB 00:00 2023-09-30T00:10:41.995 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-30T00:10:42.121 INFO:teuthology.orchestra.run.smithi040.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.7 MB/s | 384 kB 00:00 2023-09-30T00:10:42.146 INFO:teuthology.orchestra.run.smithi040.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 899 kB/s | 173 kB 00:00 2023-09-30T00:10:42.171 INFO:teuthology.orchestra.run.smithi040.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-09-30T00:10:42.205 INFO:teuthology.orchestra.run.smithi040.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 301 kB/s | 18 kB 00:00 2023-09-30T00:10:42.230 INFO:teuthology.orchestra.run.smithi040.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 831 kB/s | 49 kB 00:00 2023-09-30T00:10:42.255 INFO:teuthology.orchestra.run.smithi040.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-09-30T00:10:42.289 INFO:teuthology.orchestra.run.smithi040.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 4.7 MB/s | 283 kB 00:00 2023-09-30T00:10:42.296 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-30T00:10:42.314 INFO:teuthology.orchestra.run.smithi040.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2023-09-30T00:10:42.340 INFO:teuthology.orchestra.run.smithi040.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 374 kB/s | 19 kB 00:00 2023-09-30T00:10:42.365 INFO:teuthology.orchestra.run.smithi040.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 475 kB/s | 24 kB 00:00 2023-09-30T00:10:42.390 INFO:teuthology.orchestra.run.smithi040.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 582 kB/s | 29 kB 00:00 2023-09-30T00:10:42.415 INFO:teuthology.orchestra.run.smithi040.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 551 kB/s | 27 kB 00:00 2023-09-30T00:10:42.449 INFO:teuthology.orchestra.run.smithi040.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 4.3 MB/s | 255 kB 00:00 2023-09-30T00:10:42.474 INFO:teuthology.orchestra.run.smithi040.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.2 MB/s | 251 kB 00:00 2023-09-30T00:10:42.500 INFO:teuthology.orchestra.run.smithi040.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2023-09-30T00:10:42.516 INFO:teuthology.orchestra.run.smithi040.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 552 kB/s | 23 kB 00:00 2023-09-30T00:10:42.635 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-30T00:10:42.846 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T00:10:44.334 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T00:10:44.505 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T00:10:44.898 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T00:10:45.067 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T00:10:45.283 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T00:10:45.434 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T00:10:45.451 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T00:10:45.566 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T00:10:45.979 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T00:10:45.979 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-30T00:10:45.979 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-30T00:10:45.979 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-30T00:10:45.980 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-30T00:10:45.980 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-30T00:10:45.980 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-30T00:10:45.980 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-30T00:10:45.980 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-30T00:10:45.980 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-30T00:10:45.980 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-30T00:10:45.980 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-30T00:10:45.980 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-30T00:10:45.980 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-30T00:10:45.980 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-30T00:10:45.981 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-30T00:10:45.981 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-30T00:10:45.981 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T00:10:45.981 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-30T00:10:45.981 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-30T00:10:45.981 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-30T00:10:45.981 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-30T00:10:45.981 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-30T00:10:45.981 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-30T00:10:45.981 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-30T00:10:45.981 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-30T00:10:45.981 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-30T00:10:45.982 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-30T00:10:45.982 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-30T00:10:45.982 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-30T00:10:45.982 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-30T00:10:45.982 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-30T00:10:47.040 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-30T00:10:47.040 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:10:47.040 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:10:47.040 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:10:47.040 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:10:47.041 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:10:47.041 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:10:47.041 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:10:47.041 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:10:47.041 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T00:10:47.041 INFO:teuthology.orchestra.run.smithi135.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T00:10:47.041 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-30T00:10:47.041 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T00:10:47.041 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T00:10:47.041 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-30T00:10:47.041 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T00:10:47.041 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T00:10:47.042 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T00:10:47.042 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-30T00:10:47.042 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T00:10:47.042 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T00:10:47.042 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T00:10:47.042 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T00:10:47.042 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-30T00:10:47.042 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-30T00:10:47.042 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-30T00:10:47.042 INFO:teuthology.orchestra.run.smithi135.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T00:10:47.042 INFO:teuthology.orchestra.run.smithi135.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T00:10:47.043 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T00:10:47.043 INFO:teuthology.orchestra.run.smithi135.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T00:10:47.043 INFO:teuthology.orchestra.run.smithi135.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T00:10:47.043 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T00:10:47.043 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T00:10:47.043 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T00:10:47.043 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T00:10:47.043 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:10:47.043 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:10:47.271 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-base 2023-09-30T00:10:48.119 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:15:26 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:10:49.515 INFO:teuthology.orchestra.run.smithi135.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:10:49.701 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:10:49.702 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-09-30T00:10:49.702 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:10:49.812 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install cephadm 2023-09-30T00:10:50.651 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:15:28 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:10:52.240 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:10:52.241 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:10:52.241 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-09-30T00:10:52.241 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:10:52.241 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:10:52.241 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T00:10:52.241 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:10:52.241 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:10:52.241 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:10:52.242 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:10:52.242 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:10:52.242 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 80 k 2023-09-30T00:10:52.242 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 363 k 2023-09-30T00:10:52.242 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:10:52.492 INFO:teuthology.orchestra.run.smithi135.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 321 kB/s | 80 kB 00:00 2023-09-30T00:10:52.493 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:10:52.493 INFO:teuthology.orchestra.run.smithi135.stdout:Total 319 kB/s | 80 kB 00:00 2023-09-30T00:10:52.493 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:10:52.501 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:10:52.501 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:10:52.503 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:10:52.506 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:10:52.549 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:10:53.028 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:10:53.166 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:10:53.391 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:10:53.997 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:10:53.997 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:10:53.997 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:10:53.998 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:10:53.998 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:10:53.998 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:10:54.191 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T00:10:55.049 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:15:33 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:10:56.639 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:10:56.639 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:10:56.639 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:10:56.639 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:10:56.639 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:10:56.639 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T00:10:56.640 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:10:56.640 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:10:56.640 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:10:56.640 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:10:56.640 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:10:56.640 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 161 k 2023-09-30T00:10:56.641 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 490 k 2023-09-30T00:10:56.641 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:10:56.950 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 527 kB/s | 161 kB 00:00 2023-09-30T00:10:56.950 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:10:56.950 INFO:teuthology.orchestra.run.smithi135.stdout:Total 524 kB/s | 161 kB 00:00 2023-09-30T00:10:56.950 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:10:56.959 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:10:56.959 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:10:57.006 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:10:57.009 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:10:57.120 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:10:57.274 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:10:57.694 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:10:58.299 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:10:58.300 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:10:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:10:58.300 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:10:58.300 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:10:58.300 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:10:58.486 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr 2023-09-30T00:10:59.341 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:15:37 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:11:00.736 INFO:teuthology.orchestra.run.smithi135.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:11:00.927 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:11:00.928 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-09-30T00:11:00.928 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:11:01.034 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T00:11:01.888 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:15:39 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:11:03.480 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:11:03.481 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:11:03.481 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-09-30T00:11:03.481 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:11:03.482 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:11:03.482 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T00:11:03.482 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-09-30T00:11:03.482 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T00:11:03.482 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T00:11:03.482 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-30T00:11:03.482 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T00:11:03.482 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T00:11:03.482 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:11:03.482 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:11:03.483 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:11:03.483 INFO:teuthology.orchestra.run.smithi135.stdout:Install 6 Packages 2023-09-30T00:11:03.483 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:11:03.483 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 2.0 M 2023-09-30T00:11:03.483 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 8.0 M 2023-09-30T00:11:03.484 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:11:03.950 INFO:teuthology.orchestra.run.smithi135.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 177 kB/s | 23 kB 00:00 2023-09-30T00:11:03.975 INFO:teuthology.orchestra.run.smithi135.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 93 kB/s | 14 kB 00:00 2023-09-30T00:11:04.058 INFO:teuthology.orchestra.run.smithi135.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 393 kB/s | 43 kB 00:00 2023-09-30T00:11:04.117 INFO:teuthology.orchestra.run.smithi135.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 237 kB/s | 34 kB 00:00 2023-09-30T00:11:04.276 INFO:teuthology.orchestra.run.smithi135.stdout:(5/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 3.7 MB/s | 1.7 MB 00:00 2023-09-30T00:11:04.301 INFO:teuthology.orchestra.run.smithi135.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 811 kB/s | 196 kB 00:00 2023-09-30T00:11:04.302 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:11:04.303 INFO:teuthology.orchestra.run.smithi135.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2023-09-30T00:11:04.313 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:11:04.322 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:11:04.322 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:11:04.393 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:11:04.396 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:11:04.661 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:11:04.853 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-30T00:11:05.073 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-30T00:11:05.258 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-30T00:11:05.386 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T00:11:05.684 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T00:11:05.859 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:11:06.128 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:11:06.128 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T00:11:06.128 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T00:11:06.128 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T00:11:06.128 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-30T00:11:06.128 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-30T00:11:06.750 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-30T00:11:06.750 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:11:06.751 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:11:06.751 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:11:06.751 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:11:06.751 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:11:06.751 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T00:11:06.751 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T00:11:06.751 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T00:11:06.751 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:11:06.751 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:11:06.879 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T00:11:07.729 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:15:45 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:11:09.319 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:11:09.346 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================================ 2023-09-30T00:11:09.346 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-09-30T00:11:09.347 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================================ 2023-09-30T00:11:09.347 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:11:09.347 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T00:11:09.347 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-09-30T00:11:09.347 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-30T00:11:09.347 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-30T00:11:09.347 INFO:teuthology.orchestra.run.smithi135.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-30T00:11:09.347 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-30T00:11:09.347 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-30T00:11:09.347 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-30T00:11:09.347 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-30T00:11:09.347 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:11:09.348 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:11:09.348 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================================ 2023-09-30T00:11:09.348 INFO:teuthology.orchestra.run.smithi135.stdout:Install 8 Packages 2023-09-30T00:11:09.348 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:11:09.348 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 35 M 2023-09-30T00:11:09.348 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 210 M 2023-09-30T00:11:09.348 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:11:11.934 INFO:teuthology.orchestra.run.smithi135.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.6-116 14 MB/s | 7.4 MB 00:00 2023-09-30T00:15:16.878 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] openblas-0.3.15-4.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/openblas-0.3.15-4.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:15:16.878 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] openblas-threads-0.3.15-4.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/openblas-threads-0.3.15-4.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:15:17.225 INFO:teuthology.orchestra.run.smithi135.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 20 kB/s | 4.9 MB 04:05 2023-09-30T00:15:17.300 INFO:teuthology.orchestra.run.smithi135.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 20 kB/s | 4.7 MB 04:05 2023-09-30T00:15:33.261 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] python3-numpy-1.14.3-10.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-numpy-1.14.3-10.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:15:33.490 INFO:teuthology.orchestra.run.smithi135.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 15 kB/s | 3.7 MB 04:21 2023-09-30T00:15:33.632 INFO:teuthology.orchestra.run.smithi135.stdout:(5/8): libgfortran-8.5.0-20.el8.x86_64.rpm 4.5 MB/s | 645 kB 00:00 2023-09-30T00:15:33.674 INFO:teuthology.orchestra.run.smithi135.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 4.0 MB/s | 172 kB 00:00 2023-09-30T00:15:38.379 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] python3-pytz-2017.2-11.el8.noarch.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pytz-2017.2-11.el8.noarch.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-09-30T00:15:38.459 INFO:teuthology.orchestra.run.smithi040.stdout:(32/32): python3-pytz-2017.2-11.el8.noarch.rpm 194 B/s | 57 kB 05:01 2023-09-30T00:15:38.460 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:15:38.460 INFO:teuthology.orchestra.run.smithi040.stdout:Total 53 kB/s | 28 MB 09:07 2023-09-30T00:15:38.545 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:15:38.586 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:15:38.587 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:15:38.827 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:15:38.831 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:15:39.927 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:15:40.121 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-30T00:15:40.309 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-30T00:15:40.504 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-30T00:15:40.707 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-30T00:15:40.896 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-30T00:15:41.079 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-30T00:15:41.236 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-30T00:15:41.439 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-30T00:15:41.635 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-30T00:15:41.838 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-30T00:15:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-30T00:15:42.269 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-30T00:15:42.725 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-30T00:15:42.964 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-30T00:15:43.157 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-30T00:15:43.391 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-30T00:15:43.602 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T00:15:43.922 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-30T00:15:44.116 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-30T00:15:44.329 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-30T00:15:44.488 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-30T00:15:44.736 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-30T00:15:44.972 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-30T00:15:45.210 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-30T00:15:45.521 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-30T00:15:45.861 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-30T00:15:46.310 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-30T00:15:46.523 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T00:15:48.165 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T00:15:48.348 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T00:15:48.784 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T00:15:48.948 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T00:15:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T00:15:49.340 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T00:15:49.360 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T00:15:49.491 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T00:15:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T00:15:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-30T00:15:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-30T00:15:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-30T00:15:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-30T00:15:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-30T00:15:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-30T00:15:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-30T00:15:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-30T00:15:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-30T00:15:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-30T00:15:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-30T00:15:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-30T00:15:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-30T00:15:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-30T00:15:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-30T00:15:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-30T00:15:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T00:15:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-30T00:15:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-30T00:15:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-30T00:15:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-30T00:15:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-30T00:15:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-30T00:15:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-30T00:15:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-30T00:15:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-30T00:15:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-30T00:15:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-30T00:15:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-30T00:15:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-30T00:15:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-30T00:15:51.190 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-30T00:15:51.191 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:15:51.191 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:15:51.191 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:15:51.191 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:15:51.191 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:15:51.191 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:15:51.191 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:15:51.191 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:15:51.191 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T00:15:51.191 INFO:teuthology.orchestra.run.smithi040.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T00:15:51.192 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-30T00:15:51.192 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T00:15:51.192 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T00:15:51.192 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-30T00:15:51.192 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T00:15:51.192 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T00:15:51.192 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T00:15:51.192 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-30T00:15:51.192 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T00:15:51.192 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T00:15:51.192 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T00:15:51.192 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T00:15:51.193 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-30T00:15:51.193 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-30T00:15:51.193 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-30T00:15:51.193 INFO:teuthology.orchestra.run.smithi040.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T00:15:51.193 INFO:teuthology.orchestra.run.smithi040.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T00:15:51.193 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T00:15:51.193 INFO:teuthology.orchestra.run.smithi040.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T00:15:51.193 INFO:teuthology.orchestra.run.smithi040.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T00:15:51.193 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T00:15:51.193 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T00:15:51.193 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T00:15:51.194 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T00:15:51.194 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:15:51.194 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:15:51.342 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-base 2023-09-30T00:15:52.320 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:20:30 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:15:53.878 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:15:54.086 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:15:54.088 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-09-30T00:15:54.088 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:15:54.208 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install cephadm 2023-09-30T00:15:55.190 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:20:33 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:15:56.964 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:15:56.964 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:15:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-09-30T00:15:56.964 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:15:56.964 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:15:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T00:15:56.965 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:15:56.965 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:15:56.965 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:15:56.965 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:15:56.965 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:15:56.966 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 80 k 2023-09-30T00:15:56.966 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 363 k 2023-09-30T00:15:56.966 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:15:57.216 INFO:teuthology.orchestra.run.smithi040.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 320 kB/s | 80 kB 00:00 2023-09-30T00:15:57.216 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:15:57.217 INFO:teuthology.orchestra.run.smithi040.stdout:Total 318 kB/s | 80 kB 00:00 2023-09-30T00:15:57.217 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:15:57.227 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:15:57.227 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:15:57.230 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:15:57.233 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:15:57.277 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:15:57.793 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:15:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:15:58.194 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:15:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:15:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:15:58.845 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:15:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:15:58.845 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:15:58.845 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:15:59.065 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T00:16:00.059 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:20:38 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:16:01.811 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:16:01.812 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:16:01.812 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:16:01.812 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:16:01.812 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:16:01.812 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T00:16:01.812 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:16:01.813 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:16:01.813 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:16:01.813 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:16:01.813 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:16:01.813 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 161 k 2023-09-30T00:16:01.813 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 490 k 2023-09-30T00:16:01.813 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:16:02.095 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 574 kB/s | 161 kB 00:00 2023-09-30T00:16:02.095 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:16:02.096 INFO:teuthology.orchestra.run.smithi040.stdout:Total 571 kB/s | 161 kB 00:00 2023-09-30T00:16:02.096 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:16:02.109 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:16:02.110 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:16:02.167 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:16:02.170 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:16:02.289 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:16:02.463 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:16:02.939 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:16:03.655 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:16:03.656 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:16:03.656 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:16:03.656 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:16:03.656 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:16:03.656 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:16:03.864 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr 2023-09-30T00:16:04.845 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:20:42 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:16:06.402 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:16:06.620 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:16:06.622 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-09-30T00:16:06.622 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:16:06.739 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T00:16:07.733 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:20:45 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:16:09.490 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:16:09.491 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:16:09.491 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-09-30T00:16:09.491 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:16:09.491 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:16:09.491 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T00:16:09.491 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-09-30T00:16:09.491 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T00:16:09.491 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T00:16:09.492 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-30T00:16:09.492 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T00:16:09.492 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T00:16:09.492 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:16:09.492 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:16:09.492 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:16:09.492 INFO:teuthology.orchestra.run.smithi040.stdout:Install 6 Packages 2023-09-30T00:16:09.492 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:16:09.493 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 2.0 M 2023-09-30T00:16:09.493 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 8.0 M 2023-09-30T00:16:09.493 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:16:09.889 INFO:teuthology.orchestra.run.smithi040.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 160 kB/s | 23 kB 00:00 2023-09-30T00:16:09.907 INFO:teuthology.orchestra.run.smithi040.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 90 kB/s | 14 kB 00:00 2023-09-30T00:16:09.998 INFO:teuthology.orchestra.run.smithi040.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 393 kB/s | 43 kB 00:00 2023-09-30T00:16:10.049 INFO:teuthology.orchestra.run.smithi040.stdout:(4/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 5.6 MB/s | 1.7 MB 00:00 2023-09-30T00:16:10.074 INFO:teuthology.orchestra.run.smithi040.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 201 kB/s | 34 kB 00:00 2023-09-30T00:16:10.232 INFO:teuthology.orchestra.run.smithi040.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 841 kB/s | 196 kB 00:00 2023-09-30T00:16:10.234 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:16:10.235 INFO:teuthology.orchestra.run.smithi040.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2023-09-30T00:16:10.247 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:16:10.259 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:16:10.259 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:16:10.343 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:16:10.347 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:16:10.618 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:16:10.797 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-30T00:16:11.017 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-30T00:16:11.173 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-30T00:16:11.303 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T00:16:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T00:16:11.852 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:16:12.179 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:16:12.180 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T00:16:12.180 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T00:16:12.180 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T00:16:12.180 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-30T00:16:12.180 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-30T00:16:12.886 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-30T00:16:12.887 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:16:12.887 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:16:12.887 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:16:12.887 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:16:12.887 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:16:12.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T00:16:12.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T00:16:12.887 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T00:16:12.887 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:16:12.887 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:16:13.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T00:16:13.995 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:20:51 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:16:15.764 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:16:15.765 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================================ 2023-09-30T00:16:15.765 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-09-30T00:16:15.765 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================================ 2023-09-30T00:16:15.765 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:16:15.765 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T00:16:15.766 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-09-30T00:16:15.766 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-30T00:16:15.766 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-30T00:16:15.766 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-30T00:16:15.766 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-30T00:16:15.766 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-30T00:16:15.766 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-30T00:16:15.766 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-30T00:16:15.766 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:16:15.766 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:16:15.767 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================================ 2023-09-30T00:16:15.767 INFO:teuthology.orchestra.run.smithi040.stdout:Install 8 Packages 2023-09-30T00:16:15.767 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:16:15.767 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 35 M 2023-09-30T00:16:15.767 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 210 M 2023-09-30T00:16:15.767 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:16:16.706 INFO:teuthology.orchestra.run.smithi040.stdout:(1/8): ceph-mgr-diskprediction-local-17.2.6-116 10 MB/s | 7.4 MB 00:00 2023-09-30T00:19:22.639 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] python3-numpy-f2py-1.14.3-10.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-numpy-f2py-1.14.3-10.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:19:22.639 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-scipy-1.0.0-21.module_el8.5.0%2b771%2be5d9a225.x86_64.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:19:22.798 INFO:teuthology.orchestra.run.smithi135.stdout:(7/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 938 B/s | 225 kB 04:05 2023-09-30T00:19:23.099 INFO:teuthology.orchestra.run.smithi135.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 57 kB/s | 14 MB 04:05 2023-09-30T00:19:23.099 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:19:23.099 INFO:teuthology.orchestra.run.smithi135.stdout:Total 73 kB/s | 35 MB 08:13 2023-09-30T00:19:23.326 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:19:23.344 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:19:23.345 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:19:23.650 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:19:23.653 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:19:24.239 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:19:24.414 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T00:19:24.544 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T00:19:24.672 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T00:19:25.270 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T00:19:25.455 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:19:26.075 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:19:26.247 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T00:19:26.941 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T00:19:27.250 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T00:19:27.463 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T00:19:29.210 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T00:19:30.486 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-30T00:19:30.721 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:19:31.027 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:19:31.027 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T00:19:31.027 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-30T00:19:31.027 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:19:31.027 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-30T00:19:31.028 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T00:19:31.028 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-30T00:19:31.028 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-30T00:19:31.680 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-30T00:19:31.681 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:19:31.681 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:19:31.681 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:19:31.681 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-30T00:19:31.681 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-30T00:19:31.681 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-30T00:19:31.682 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-30T00:19:31.682 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-30T00:19:31.682 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-30T00:19:31.682 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-30T00:19:31.682 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:19:31.682 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:19:31.800 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-rook 2023-09-30T00:19:32.660 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:24:10 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:19:34.243 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:19:34.245 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:19:34.245 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-09-30T00:19:34.245 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:19:34.245 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:19:34.245 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T00:19:34.246 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-09-30T00:19:34.246 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T00:19:34.246 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T00:19:34.246 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T00:19:34.246 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-30T00:19:34.246 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-30T00:19:34.246 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T00:19:34.246 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-30T00:19:34.246 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-30T00:19:34.246 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-30T00:19:34.247 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-30T00:19:34.247 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T00:19:34.247 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T00:19:34.247 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:19:34.247 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:19:34.247 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:19:34.247 INFO:teuthology.orchestra.run.smithi135.stdout:Install 13 Packages 2023-09-30T00:19:34.247 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:19:34.248 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 1.9 M 2023-09-30T00:19:34.248 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 25 M 2023-09-30T00:19:34.248 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:19:34.880 INFO:teuthology.orchestra.run.smithi135.stdout:(1/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 117 kB/s | 47 kB 00:00 2023-09-30T00:20:27.145 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] openblas-0.3.15-4.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/openblas-0.3.15-4.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:20:27.145 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] openblas-threads-0.3.15-4.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/openblas-threads-0.3.15-4.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:20:27.358 INFO:teuthology.orchestra.run.smithi040.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 19 kB/s | 4.7 MB 04:11 2023-09-30T00:20:27.450 INFO:teuthology.orchestra.run.smithi040.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 20 kB/s | 4.9 MB 04:11 2023-09-30T00:21:17.599 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] python3-numpy-1.14.3-10.el8.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-numpy-1.14.3-10.el8.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-09-30T00:21:18.144 INFO:teuthology.orchestra.run.smithi040.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 13 kB/s | 3.7 MB 05:01 2023-09-30T00:21:18.269 INFO:teuthology.orchestra.run.smithi040.stdout:(5/8): libgfortran-8.5.0-20.el8.x86_64.rpm 5.0 MB/s | 645 kB 00:00 2023-09-30T00:21:18.311 INFO:teuthology.orchestra.run.smithi040.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 4.1 MB/s | 172 kB 00:00 2023-09-30T00:23:40.873 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] python3-jsonpatch-1.21-2.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-jsonpatch-1.21-2.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:23:40.873 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] python3-jsonpointer-1.10-11.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-jsonpointer-1.10-11.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:23:40.982 INFO:teuthology.orchestra.run.smithi135.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 83 B/s | 20 kB 04:06 2023-09-30T00:23:40.999 INFO:teuthology.orchestra.run.smithi135.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 110 B/s | 27 kB 04:06 2023-09-30T00:23:41.107 INFO:teuthology.orchestra.run.smithi135.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.4 MB/s | 155 kB 00:00 2023-09-30T00:23:41.141 INFO:teuthology.orchestra.run.smithi135.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.3 MB/s | 43 kB 00:00 2023-09-30T00:23:41.241 INFO:teuthology.orchestra.run.smithi135.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 330 kB/s | 33 kB 00:00 2023-09-30T00:23:41.291 INFO:teuthology.orchestra.run.smithi135.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 327 kB/s | 16 kB 00:00 2023-09-30T00:23:41.392 INFO:teuthology.orchestra.run.smithi135.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 812 kB/s | 81 kB 00:00 2023-09-30T00:23:41.517 INFO:teuthology.orchestra.run.smithi135.stdout:(9/13): python3-kubernetes-11.0.0-6.el8.noarch. 9.3 MB/s | 1.2 MB 00:00 2023-09-30T00:23:41.567 INFO:teuthology.orchestra.run.smithi135.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2023-09-30T00:23:41.634 INFO:teuthology.orchestra.run.smithi135.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 918 kB/s | 61 kB 00:00 2023-09-30T00:24:32.880 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] python3-numpy-f2py-1.14.3-10.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-numpy-f2py-1.14.3-10.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:24:33.184 INFO:teuthology.orchestra.run.smithi040.stdout:(7/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 937 B/s | 225 kB 04:05 2023-09-30T00:24:50.887 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] python3-pyasn1-0.3.7-6.el8.noarch.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pyasn1-0.3.7-6.el8.noarch.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-09-30T00:24:51.216 INFO:teuthology.orchestra.run.smithi135.stdout:(12/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 409 B/s | 126 kB 05:16 2023-09-30T00:25:28.452 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-scipy-1.0.0-21.module_el8.5.0%2b771%2be5d9a225.x86_64.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-09-30T00:25:29.202 INFO:teuthology.orchestra.run.smithi040.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 46 kB/s | 14 MB 05:01 2023-09-30T00:25:29.203 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:25:29.203 INFO:teuthology.orchestra.run.smithi040.stdout:Total 65 kB/s | 35 MB 09:13 2023-09-30T00:25:29.446 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:25:29.467 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:25:29.467 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:25:29.805 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:25:29.809 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:25:30.400 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:25:30.576 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T00:25:30.716 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T00:25:30.850 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T00:25:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T00:25:31.684 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:25:32.362 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:25:32.517 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T00:25:33.271 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T00:25:33.582 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T00:25:33.808 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T00:25:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T00:25:37.091 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-30T00:25:37.316 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:25:37.673 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:25:37.673 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T00:25:37.674 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-30T00:25:37.674 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:25:37.674 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-30T00:25:37.674 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T00:25:37.674 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-30T00:25:37.674 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-30T00:25:38.396 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-30T00:25:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:25:38.397 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:25:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:25:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-30T00:25:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-30T00:25:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-30T00:25:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-30T00:25:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-30T00:25:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-30T00:25:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-30T00:25:38.397 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:25:38.398 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:25:38.529 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-rook 2023-09-30T00:25:39.517 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:30:17 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:25:41.271 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:25:41.273 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:25:41.273 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-09-30T00:25:41.273 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:25:41.273 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:25:41.274 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T00:25:41.274 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-09-30T00:25:41.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T00:25:41.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T00:25:41.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T00:25:41.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-30T00:25:41.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-30T00:25:41.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T00:25:41.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-30T00:25:41.274 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-30T00:25:41.275 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-30T00:25:41.275 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-30T00:25:41.275 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T00:25:41.275 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T00:25:41.275 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:25:41.275 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:25:41.275 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:25:41.275 INFO:teuthology.orchestra.run.smithi040.stdout:Install 13 Packages 2023-09-30T00:25:41.275 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:25:41.276 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 1.9 M 2023-09-30T00:25:41.276 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 25 M 2023-09-30T00:25:41.276 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:25:41.776 INFO:teuthology.orchestra.run.smithi040.stdout:(1/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 210 kB/s | 47 kB 00:00 2023-09-30T00:27:54.638 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] python3-pyasn1-modules-0.3.7-6.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pyasn1-modules-0.3.7-6.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:27:54.989 INFO:teuthology.orchestra.run.smithi135.stdout:(13/13): python3-pyasn1-modules-0.3.7-6.el8.noa 443 B/s | 110 kB 04:14 2023-09-30T00:27:54.989 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:27:54.989 INFO:teuthology.orchestra.run.smithi135.stdout:Total 3.9 kB/s | 1.9 MB 08:20 2023-09-30T00:27:55.043 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:27:55.060 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:27:55.060 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:27:55.147 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:27:55.150 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:27:55.671 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:27:55.906 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T00:27:56.086 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-30T00:27:56.289 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-30T00:27:56.527 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-30T00:27:56.742 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-30T00:27:56.965 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-30T00:27:57.240 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-30T00:27:57.458 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T00:27:58.470 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T00:27:58.743 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-30T00:27:58.944 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T00:27:59.145 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-30T00:27:59.303 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:27:59.596 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:27:59.596 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T00:27:59.596 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-30T00:27:59.596 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-30T00:27:59.596 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-30T00:27:59.596 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-30T00:27:59.596 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T00:27:59.596 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T00:27:59.596 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-30T00:27:59.597 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T00:27:59.597 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-30T00:27:59.597 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-30T00:27:59.597 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-30T00:28:00.291 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-30T00:28:00.291 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:28:00.291 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:28:00.292 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:28:00.292 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T00:28:00.292 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T00:28:00.292 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T00:28:00.292 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T00:28:00.292 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T00:28:00.292 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T00:28:00.292 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T00:28:00.292 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T00:28:00.292 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T00:28:00.292 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T00:28:00.293 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T00:28:00.293 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T00:28:00.293 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:28:00.293 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:28:00.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T00:28:01.264 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:32:39 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:28:02.850 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:28:02.851 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:28:02.851 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-09-30T00:28:02.851 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:28:02.851 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:28:02.851 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T00:28:02.851 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-09-30T00:28:02.851 INFO:teuthology.orchestra.run.smithi135.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T00:28:02.851 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-30T00:28:02.852 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-30T00:28:02.852 INFO:teuthology.orchestra.run.smithi135.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T00:28:02.852 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:28:02.852 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:28:02.852 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:28:02.852 INFO:teuthology.orchestra.run.smithi135.stdout:Install 5 Packages 2023-09-30T00:28:02.852 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:28:02.853 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 6.0 M 2023-09-30T00:28:02.853 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 26 M 2023-09-30T00:28:02.853 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:28:03.855 INFO:teuthology.orchestra.run.smithi135.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 655 kB/s | 121 kB 00:00 2023-09-30T00:28:04.530 INFO:teuthology.orchestra.run.smithi135.stdout:(2/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 705 kB/s | 476 kB 00:00 2023-09-30T00:28:04.638 INFO:teuthology.orchestra.run.smithi135.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 556 kB/s | 60 kB 00:00 2023-09-30T00:29:56.464 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] python3-jsonpointer-1.10-11.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-jsonpointer-1.10-11.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:29:56.465 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] python3-jsonpatch-1.21-2.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-jsonpatch-1.21-2.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:29:56.529 INFO:teuthology.orchestra.run.smithi040.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 80 B/s | 20 kB 04:14 2023-09-30T00:29:56.554 INFO:teuthology.orchestra.run.smithi040.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 107 B/s | 27 kB 04:15 2023-09-30T00:29:56.654 INFO:teuthology.orchestra.run.smithi040.stdout:(4/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.5 MB/s | 155 kB 00:00 2023-09-30T00:29:56.688 INFO:teuthology.orchestra.run.smithi040.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.2 MB/s | 43 kB 00:00 2023-09-30T00:29:56.922 INFO:teuthology.orchestra.run.smithi040.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 141 kB/s | 33 kB 00:00 2023-09-30T00:29:57.014 INFO:teuthology.orchestra.run.smithi040.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 177 kB/s | 16 kB 00:00 2023-09-30T00:29:57.181 INFO:teuthology.orchestra.run.smithi040.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 488 kB/s | 81 kB 00:00 2023-09-30T00:29:57.498 INFO:teuthology.orchestra.run.smithi040.stdout:(9/13): python3-kubernetes-11.0.0-6.el8.noarch. 3.7 MB/s | 1.2 MB 00:00 2023-09-30T00:29:57.590 INFO:teuthology.orchestra.run.smithi040.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 700 kB/s | 64 kB 00:00 2023-09-30T00:29:57.682 INFO:teuthology.orchestra.run.smithi040.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 667 kB/s | 61 kB 00:00 2023-09-30T00:30:04.758 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] python3-pyasn1-0.3.7-6.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pyasn1-0.3.7-6.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:30:04.831 INFO:teuthology.orchestra.run.smithi040.stdout:(12/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 492 B/s | 126 kB 04:23 2023-09-30T00:32:13.062 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] python3-babel-2.5.1-7.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-babel-2.5.1-7.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:32:13.306 INFO:teuthology.orchestra.run.smithi135.stdout:(4/5): python3-babel-2.5.1-7.el8.noarch.rpm 20 kB/s | 4.8 MB 04:09 2023-09-30T00:33:04.673 INFO:teuthology.orchestra.run.smithi135.stdout:[MIRROR] python3-jinja2-2.10.1-3.el8.noarch.rpm: Curl error (28): Timeout was reached for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.10.1-3.el8.noarch.rpm [Operation too slow. Less than 1000 bytes/sec transferred the last 300 seconds] 2023-09-30T00:33:05.107 INFO:teuthology.orchestra.run.smithi135.stdout:(5/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.8 kB/s | 538 kB 05:01 2023-09-30T00:33:05.107 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:33:05.107 INFO:teuthology.orchestra.run.smithi135.stdout:Total 20 kB/s | 6.0 MB 05:02 2023-09-30T00:33:05.160 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:33:05.172 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:33:05.172 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:33:05.267 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:33:05.270 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:33:05.577 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:33:05.879 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-30T00:33:06.813 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T00:33:07.144 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-30T00:33:07.410 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-30T00:33:07.582 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:33:07.868 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:33:07.869 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T00:33:07.869 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-30T00:33:07.869 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-30T00:33:07.869 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T00:33:08.503 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-30T00:33:08.503 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:08.503 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:33:08.503 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:33:08.503 INFO:teuthology.orchestra.run.smithi135.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T00:33:08.503 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-30T00:33:08.503 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-30T00:33:08.503 INFO:teuthology.orchestra.run.smithi135.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T00:33:08.503 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:08.504 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:33:08.624 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-fuse 2023-09-30T00:33:09.478 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:37:47 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:33:11.041 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:33:11.042 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:11.042 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-09-30T00:33:11.042 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:11.042 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:33:11.042 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T00:33:11.042 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-09-30T00:33:11.042 INFO:teuthology.orchestra.run.smithi135.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-30T00:33:11.042 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:11.043 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:33:11.043 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:11.043 INFO:teuthology.orchestra.run.smithi135.stdout:Install 2 Packages 2023-09-30T00:33:11.043 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:11.043 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 957 k 2023-09-30T00:33:11.043 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 3.0 M 2023-09-30T00:33:11.044 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:33:11.948 INFO:teuthology.orchestra.run.smithi135.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 407 kB/s | 83 kB 00:00 2023-09-30T00:33:12.507 INFO:teuthology.orchestra.run.smithi135.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 1.1 MB/s | 874 kB 00:00 2023-09-30T00:33:12.507 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:33:12.508 INFO:teuthology.orchestra.run.smithi135.stdout:Total 654 kB/s | 957 kB 00:01 2023-09-30T00:33:12.519 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:33:12.542 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:33:12.542 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:33:12.634 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:33:12.637 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:33:12.804 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:33:13.037 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-30T00:33:13.232 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:33:13.625 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:33:13.626 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T00:33:14.241 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-30T00:33:14.241 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:14.242 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:33:14.242 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-30T00:33:14.242 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:14.242 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:33:14.443 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-volume 2023-09-30T00:33:15.309 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:37:53 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:33:16.895 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:33:16.896 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:16.896 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-09-30T00:33:16.896 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:16.896 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:33:16.896 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T00:33:16.897 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:16.897 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:33:16.897 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:16.897 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:33:16.897 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:16.897 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 267 k 2023-09-30T00:33:16.898 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 1.3 M 2023-09-30T00:33:16.898 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:33:17.552 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 407 kB/s | 267 kB 00:00 2023-09-30T00:33:17.553 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:33:17.553 INFO:teuthology.orchestra.run.smithi135.stdout:Total 407 kB/s | 267 kB 00:00 2023-09-30T00:33:17.553 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:33:17.563 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:33:17.564 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:33:17.592 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:33:17.595 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:33:17.771 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:33:17.998 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:33:18.422 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:33:19.033 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:33:19.034 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:19.034 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:33:19.034 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:33:19.034 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:19.034 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:33:19.233 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados-devel 2023-09-30T00:33:20.098 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:37:58 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:33:21.679 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:33:21.679 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:21.679 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:33:21.679 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:21.680 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:33:21.680 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T00:33:21.680 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:21.680 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:33:21.680 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:21.680 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:33:21.680 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:21.681 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 130 k 2023-09-30T00:33:21.681 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 464 k 2023-09-30T00:33:21.681 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:33:22.107 INFO:teuthology.orchestra.run.smithi135.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 305 kB/s | 130 kB 00:00 2023-09-30T00:33:22.108 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:33:22.108 INFO:teuthology.orchestra.run.smithi135.stdout:Total 304 kB/s | 130 kB 00:00 2023-09-30T00:33:22.108 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:33:22.113 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:33:22.113 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:33:22.162 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:33:22.165 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:33:22.273 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:33:22.496 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:33:22.778 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:33:23.393 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:33:23.393 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:23.393 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:33:23.394 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:33:23.394 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:23.394 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:33:23.583 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install libcephfs2 2023-09-30T00:33:24.443 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:38:02 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:33:25.838 INFO:teuthology.orchestra.run.smithi135.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:33:26.025 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:33:26.026 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-09-30T00:33:26.026 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:33:26.130 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install libcephfs-devel 2023-09-30T00:33:26.987 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:38:04 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:33:28.557 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:33:28.558 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:28.558 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:33:28.558 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:28.558 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:33:28.558 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T00:33:28.558 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:28.558 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:33:28.558 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:28.558 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:33:28.558 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:28.559 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 25 k 2023-09-30T00:33:28.559 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 109 k 2023-09-30T00:33:28.560 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:33:28.802 INFO:teuthology.orchestra.run.smithi135.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 102 kB/s | 25 kB 00:00 2023-09-30T00:33:28.802 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:33:28.802 INFO:teuthology.orchestra.run.smithi135.stdout:Total 102 kB/s | 25 kB 00:00 2023-09-30T00:33:28.803 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:33:28.806 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:33:28.806 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:33:28.808 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:33:28.811 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:33:28.858 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:33:29.053 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:33:29.317 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:33:29.901 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:33:29.902 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:29.902 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:33:29.902 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:33:29.902 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:29.902 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:33:30.012 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados2 2023-09-30T00:33:30.865 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:38:08 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:33:32.251 INFO:teuthology.orchestra.run.smithi135.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:33:32.438 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:33:32.439 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-09-30T00:33:32.439 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:33:32.543 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librbd1 2023-09-30T00:33:33.405 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:38:11 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:33:34.805 INFO:teuthology.orchestra.run.smithi135.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:33:34.993 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:33:34.994 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-09-30T00:33:34.995 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:33:35.091 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rados 2023-09-30T00:33:35.952 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:38:13 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:33:37.344 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:33:37.532 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:33:37.533 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-09-30T00:33:37.533 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:33:37.642 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rgw 2023-09-30T00:33:38.502 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:38:16 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:33:39.894 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:33:40.081 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:33:40.082 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-09-30T00:33:40.083 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:33:40.191 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-cephfs 2023-09-30T00:33:41.049 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:38:19 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:33:42.442 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:33:42.631 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:33:42.632 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-09-30T00:33:42.632 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:33:42.738 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rbd 2023-09-30T00:33:43.590 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:38:21 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:33:45.011 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:33:45.198 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:33:45.199 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-09-30T00:33:45.200 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:33:45.309 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-fuse 2023-09-30T00:33:46.163 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:38:24 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:33:47.841 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:33:47.842 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:47.842 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2023-09-30T00:33:47.842 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:47.842 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:33:47.842 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T00:33:47.843 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:47.843 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:33:47.843 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:47.843 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:33:47.843 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:47.844 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 90 k 2023-09-30T00:33:47.844 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 264 k 2023-09-30T00:33:47.844 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:33:48.372 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 171 kB/s | 90 kB 00:00 2023-09-30T00:33:48.373 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:33:48.373 INFO:teuthology.orchestra.run.smithi135.stdout:Total 170 kB/s | 90 kB 00:00 2023-09-30T00:33:48.373 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:33:48.383 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:33:48.383 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:33:48.434 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:33:48.437 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:33:48.540 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:33:48.742 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:33:49.016 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:33:49.613 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:33:49.613 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:49.613 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:33:49.613 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:33:49.614 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:49.614 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:33:49.832 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-mirror 2023-09-30T00:33:50.694 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:38:28 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:33:52.279 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:33:52.280 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:52.280 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:33:52.280 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:52.281 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:33:52.281 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:33:52.281 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:52.281 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:33:52.281 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:52.281 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:33:52.281 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:52.281 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 3.5 M 2023-09-30T00:33:52.281 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 16 M 2023-09-30T00:33:52.281 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:33:55.358 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 1.1 MB/s | 3.5 MB 00:03 2023-09-30T00:33:55.359 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:33:55.360 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.1 MB/s | 3.5 MB 00:03 2023-09-30T00:33:55.360 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:33:55.373 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:33:55.373 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:33:55.440 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:33:55.443 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:33:55.881 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:33:56.076 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:33:56.485 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:33:57.100 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:33:57.100 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:57.100 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:33:57.101 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:33:57.101 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:57.101 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:33:57.319 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-nbd 2023-09-30T00:33:58.173 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:38:36 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:33:59.757 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:33:59.758 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:59.758 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2023-09-30T00:33:59.758 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:59.758 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:33:59.758 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T00:33:59.758 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:59.758 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:33:59.758 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:33:59.759 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:33:59.759 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:33:59.759 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 173 k 2023-09-30T00:33:59.759 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 552 k 2023-09-30T00:33:59.759 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:34:00.211 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 382 kB/s | 173 kB 00:00 2023-09-30T00:34:00.212 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:34:00.212 INFO:teuthology.orchestra.run.smithi135.stdout:Total 381 kB/s | 173 kB 00:00 2023-09-30T00:34:00.212 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:34:00.223 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:34:00.223 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:34:00.273 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:34:00.276 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:34:00.396 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:34:00.565 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:34:00.865 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:34:01.479 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:34:01.479 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:34:01.480 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:34:01.480 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:34:01.480 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:34:01.480 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:34:01.697 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-base-debuginfo 2023-09-30T00:34:02.223 INFO:teuthology.orchestra.run.smithi040.stdout:[MIRROR] python3-pyasn1-modules-0.3.7-6.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/AppStream/x86_64/os/Packages/python3-pyasn1-modules-0.3.7-6.el8.noarch.rpm [Recv failure: Connection reset by peer] 2023-09-30T00:34:02.332 INFO:teuthology.orchestra.run.smithi040.stdout:(13/13): python3-pyasn1-modules-0.3.7-6.el8.noa 458 B/s | 110 kB 04:05 2023-09-30T00:34:02.332 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:34:02.333 INFO:teuthology.orchestra.run.smithi040.stdout:Total 3.9 kB/s | 1.9 MB 08:21 2023-09-30T00:34:02.391 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:34:02.409 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:34:02.409 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:34:02.506 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:34:02.509 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:34:02.556 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:38:40 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:03.006 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:34:03.356 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T00:34:03.535 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-30T00:34:03.745 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-30T00:34:03.954 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-30T00:34:04.112 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:34:04.113 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:34:04.113 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:34:04.113 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:34:04.113 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:34:04.113 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 86 M 2023-09-30T00:34:04.114 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-09-30T00:34:04.114 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 M 2023-09-30T00:34:04.114 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:34:04.114 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:34:04.114 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:34:04.114 INFO:teuthology.orchestra.run.smithi135.stdout:Install 2 Packages 2023-09-30T00:34:04.114 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:34:04.115 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 110 M 2023-09-30T00:34:04.115 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 659 M 2023-09-30T00:34:04.115 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:34:04.139 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-30T00:34:04.356 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-30T00:34:04.627 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-30T00:34:04.855 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T00:34:05.973 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T00:34:06.238 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-30T00:34:06.421 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T00:34:06.641 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-30T00:34:06.795 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:34:07.133 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:34:07.134 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T00:34:07.134 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-30T00:34:07.134 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-30T00:34:07.134 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-30T00:34:07.134 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-30T00:34:07.134 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T00:34:07.134 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T00:34:07.135 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-30T00:34:07.135 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T00:34:07.135 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-30T00:34:07.135 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-30T00:34:07.135 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-30T00:34:07.895 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-30T00:34:07.895 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:07.895 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:34:07.895 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:34:07.895 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T00:34:07.895 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T00:34:07.896 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T00:34:07.896 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T00:34:07.896 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T00:34:07.896 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T00:34:07.896 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T00:34:07.896 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T00:34:07.896 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T00:34:07.896 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T00:34:07.896 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T00:34:07.896 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T00:34:07.896 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:07.897 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:34:08.026 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T00:34:08.097 INFO:teuthology.orchestra.run.smithi135.stdout:(1/2): ceph-debuginfo-17.2.6-1165.g93dce5e1.el8 6.0 MB/s | 24 MB 00:03 2023-09-30T00:34:09.017 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:38:47 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:09.639 INFO:teuthology.orchestra.run.smithi135.stdout:(2/2): ceph-base-debuginfo-17.2.6-1165.g93dce5e 16 MB/s | 86 MB 00:05 2023-09-30T00:34:09.641 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:34:09.641 INFO:teuthology.orchestra.run.smithi135.stdout:Total 20 MB/s | 110 MB 00:05 2023-09-30T00:34:09.641 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:34:09.646 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:34:09.646 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:34:10.114 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:34:10.117 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:34:10.790 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:34:10.791 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:10.791 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-09-30T00:34:10.791 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:10.791 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:34:10.791 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T00:34:10.792 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-09-30T00:34:10.792 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T00:34:10.792 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-30T00:34:10.792 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-30T00:34:10.792 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T00:34:10.792 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:10.793 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:34:10.793 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:10.793 INFO:teuthology.orchestra.run.smithi040.stdout:Install 5 Packages 2023-09-30T00:34:10.793 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:10.794 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 6.0 M 2023-09-30T00:34:10.794 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 26 M 2023-09-30T00:34:10.794 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:34:11.199 INFO:teuthology.orchestra.run.smithi040.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 643 kB/s | 121 kB 00:00 2023-09-30T00:34:11.299 INFO:teuthology.orchestra.run.smithi040.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.8 MB/s | 538 kB 00:00 2023-09-30T00:34:11.616 INFO:teuthology.orchestra.run.smithi040.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 7.9 MB/s | 4.8 MB 00:00 2023-09-30T00:34:12.008 INFO:teuthology.orchestra.run.smithi040.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 85 kB/s | 60 kB 00:00 2023-09-30T00:34:12.291 INFO:teuthology.orchestra.run.smithi040.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 436 kB/s | 476 kB 00:01 2023-09-30T00:34:12.292 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:34:12.292 INFO:teuthology.orchestra.run.smithi040.stdout:Total 4.0 MB/s | 6.0 MB 00:01 2023-09-30T00:34:12.348 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:34:12.361 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:34:12.361 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:34:12.466 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:34:12.470 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:34:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:34:13.084 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-30T00:34:13.115 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:34:14.123 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T00:34:14.500 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-30T00:34:14.728 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-30T00:34:14.903 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:34:15.238 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:34:15.239 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T00:34:15.239 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-30T00:34:15.239 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-30T00:34:15.239 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T00:34:15.938 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-30T00:34:15.938 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:15.938 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:34:15.939 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:34:15.939 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T00:34:15.939 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-30T00:34:15.939 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-30T00:34:15.939 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T00:34:15.939 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:15.939 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:34:16.058 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-fuse 2023-09-30T00:34:17.051 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:38:55 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:18.801 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:34:18.802 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:18.802 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-09-30T00:34:18.802 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:18.802 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:34:18.802 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T00:34:18.802 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-09-30T00:34:18.802 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-30T00:34:18.802 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:18.802 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:34:18.803 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:18.803 INFO:teuthology.orchestra.run.smithi040.stdout:Install 2 Packages 2023-09-30T00:34:18.803 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:18.803 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 957 k 2023-09-30T00:34:18.803 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 3.0 M 2023-09-30T00:34:18.804 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:34:19.207 INFO:teuthology.orchestra.run.smithi040.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 375 kB/s | 83 kB 00:00 2023-09-30T00:34:19.516 INFO:teuthology.orchestra.run.smithi040.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 1.6 MB/s | 874 kB 00:00 2023-09-30T00:34:19.516 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:34:19.517 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.3 MB/s | 957 kB 00:00 2023-09-30T00:34:19.529 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:34:19.554 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:34:19.554 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:34:19.657 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:34:19.660 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:34:19.841 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:34:20.116 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-30T00:34:20.329 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:34:20.785 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:34:20.785 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T00:34:21.512 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-30T00:34:21.512 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:21.512 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:34:21.512 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-30T00:34:21.512 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:21.512 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:34:21.742 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-volume 2023-09-30T00:34:22.733 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:39:00 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:22.745 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T00:34:22.991 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 2/2 2023-09-30T00:34:23.236 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 2/2 2023-09-30T00:34:23.237 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/2 2023-09-30T00:34:23.845 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:34:23.845 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:34:23.845 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:34:23.845 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:34:23.845 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:34:23.845 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:34:23.845 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:34:23.966 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-common-debuginfo 2023-09-30T00:34:24.497 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:34:24.498 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:24.498 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2023-09-30T00:34:24.498 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:24.498 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:34:24.498 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T00:34:24.498 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:24.498 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:34:24.498 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:24.498 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:34:24.499 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:24.499 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 267 k 2023-09-30T00:34:24.499 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 1.3 M 2023-09-30T00:34:24.499 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:34:24.711 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 1.2 MB/s | 267 kB 00:00 2023-09-30T00:34:24.711 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:34:24.711 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.2 MB/s | 267 kB 00:00 2023-09-30T00:34:24.712 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:34:24.724 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:34:24.724 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:34:24.758 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:34:24.761 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:34:24.824 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:39:02 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:24.957 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:34:25.160 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:34:25.617 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:34:26.330 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:34:26.330 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:26.330 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:34:26.330 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:34:26.330 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:26.330 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:34:26.428 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:34:26.429 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:34:26.429 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:34:26.429 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:34:26.429 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:34:26.429 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 557 M 2023-09-30T00:34:26.429 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:34:26.429 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:34:26.429 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:34:26.430 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:34:26.430 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:34:26.430 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 557 M 2023-09-30T00:34:26.430 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 2.9 G 2023-09-30T00:34:26.431 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:34:26.532 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados-devel 2023-09-30T00:34:27.535 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:39:05 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:29.290 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:34:29.291 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:34:29.291 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:29.291 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:34:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T00:34:29.291 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:29.291 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:34:29.292 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:29.292 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:34:29.292 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:29.292 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 130 k 2023-09-30T00:34:29.292 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 464 k 2023-09-30T00:34:29.292 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:34:29.798 INFO:teuthology.orchestra.run.smithi040.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 257 kB/s | 130 kB 00:00 2023-09-30T00:34:29.798 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:34:29.799 INFO:teuthology.orchestra.run.smithi040.stdout:Total 256 kB/s | 130 kB 00:00 2023-09-30T00:34:29.799 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:34:29.806 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:34:29.806 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:34:29.866 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:34:29.869 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:34:29.992 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:34:30.218 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:34:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:34:31.261 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:34:31.262 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:31.262 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:34:31.262 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:34:31.262 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:31.262 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:34:31.485 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs2 2023-09-30T00:34:32.487 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:39:10 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:34.024 INFO:teuthology.orchestra.run.smithi040.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:34:34.233 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:34:34.234 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-09-30T00:34:34.235 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:34:34.337 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs-devel 2023-09-30T00:34:35.328 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:39:13 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:37.090 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:34:37.094 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:37.094 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:34:37.094 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:37.094 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:34:37.094 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T00:34:37.094 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:37.094 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:34:37.094 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:37.095 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:34:37.095 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:37.095 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 25 k 2023-09-30T00:34:37.095 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 109 k 2023-09-30T00:34:37.095 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:34:37.242 INFO:teuthology.orchestra.run.smithi040.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 166 kB/s | 25 kB 00:00 2023-09-30T00:34:37.243 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:34:37.243 INFO:teuthology.orchestra.run.smithi040.stdout:Total 164 kB/s | 25 kB 00:00 2023-09-30T00:34:37.243 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:34:37.247 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:34:37.247 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:34:37.250 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:34:37.253 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:34:37.308 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:34:37.516 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:34:37.801 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:34:38.468 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:34:38.468 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:38.468 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:34:38.468 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:34:38.468 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:38.468 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:34:38.584 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados2 2023-09-30T00:34:39.572 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:39:17 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:41.124 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:34:41.334 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:34:41.335 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-09-30T00:34:41.335 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:34:41.438 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librbd1 2023-09-30T00:34:42.427 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:39:20 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:43.973 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:34:44.183 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:34:44.184 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-09-30T00:34:44.185 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:34:44.295 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rados 2023-09-30T00:34:45.281 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:39:23 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:46.839 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:34:47.049 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:34:47.050 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-09-30T00:34:47.050 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:34:47.165 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rgw 2023-09-30T00:34:48.151 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:39:26 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:49.705 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:34:49.914 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:34:49.915 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-09-30T00:34:49.915 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:34:50.026 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-cephfs 2023-09-30T00:34:51.016 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:39:29 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:52.568 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:34:52.778 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:34:52.779 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-09-30T00:34:52.779 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:34:52.883 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rbd 2023-09-30T00:34:53.873 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:39:31 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:55.427 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:34:55.638 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:34:55.639 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-09-30T00:34:55.639 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:34:55.744 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-fuse 2023-09-30T00:34:56.726 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:39:34 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:34:58.498 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:34:58.499 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:58.499 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2023-09-30T00:34:58.499 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:58.499 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:34:58.499 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T00:34:58.499 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:58.499 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:34:58.499 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:34:58.500 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:34:58.500 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:34:58.500 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 90 k 2023-09-30T00:34:58.500 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 264 k 2023-09-30T00:34:58.500 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:34:58.940 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 205 kB/s | 90 kB 00:00 2023-09-30T00:34:58.941 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:34:58.941 INFO:teuthology.orchestra.run.smithi040.stdout:Total 204 kB/s | 90 kB 00:00 2023-09-30T00:34:58.941 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:34:58.954 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:34:58.954 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:34:59.015 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:34:59.018 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:34:59.156 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:34:59.388 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:34:59.714 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:35:00.411 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:35:00.411 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:00.411 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:35:00.411 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:00.411 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:00.412 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:35:00.621 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-mirror 2023-09-30T00:35:01.606 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:39:39 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:35:01.624 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-common-debuginfo-17.2.6-1165.g93dce5e1.el8 16 MB/s | 557 MB 00:35 2023-09-30T00:35:01.625 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:35:01.625 INFO:teuthology.orchestra.run.smithi135.stdout:Total 16 MB/s | 557 MB 00:35 2023-09-30T00:35:01.626 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:35:01.630 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:35:01.630 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:35:03.364 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:35:03.365 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:35:03.365 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:35:03.365 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:35:03.365 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:35:03.365 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:35:03.365 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:03.365 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:35:03.365 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:35:03.365 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:35:03.365 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:03.366 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 3.5 M 2023-09-30T00:35:03.366 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 16 M 2023-09-30T00:35:03.366 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:35:03.754 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:35:03.756 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:35:03.835 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 7.4 MB/s | 3.5 MB 00:00 2023-09-30T00:35:03.836 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:35:03.837 INFO:teuthology.orchestra.run.smithi040.stdout:Total 7.3 MB/s | 3.5 MB 00:00 2023-09-30T00:35:03.837 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:35:03.851 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:35:03.852 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:35:03.925 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:35:03.928 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:35:04.407 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:35:04.588 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:35:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:35:05.793 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:35:05.793 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:05.793 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:35:05.793 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:05.793 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:05.793 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:35:05.996 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-nbd 2023-09-30T00:35:06.984 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:39:44 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:35:08.744 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:35:08.745 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:35:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2023-09-30T00:35:08.745 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:35:08.745 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:35:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T00:35:08.745 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:08.746 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:35:08.746 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:35:08.746 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:35:08.746 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:08.746 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 173 k 2023-09-30T00:35:08.747 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 552 k 2023-09-30T00:35:08.747 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:35:08.930 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 944 kB/s | 173 kB 00:00 2023-09-30T00:35:08.930 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:35:08.931 INFO:teuthology.orchestra.run.smithi040.stdout:Total 936 kB/s | 173 kB 00:00 2023-09-30T00:35:08.931 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:35:08.944 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:35:08.944 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:35:09.002 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:35:09.006 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:35:09.141 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:35:09.596 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:35:09.914 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:35:10.610 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:35:10.610 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:10.611 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:35:10.611 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:10.611 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:10.611 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:35:10.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-base-debuginfo 2023-09-30T00:35:11.787 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:39:49 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:35:13.548 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:35:13.549 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:35:13.549 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:35:13.549 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:35:13.549 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:35:13.549 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 86 M 2023-09-30T00:35:13.549 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2023-09-30T00:35:13.549 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 M 2023-09-30T00:35:13.549 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:13.549 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:35:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:35:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:Install 2 Packages 2023-09-30T00:35:13.550 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 110 M 2023-09-30T00:35:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 659 M 2023-09-30T00:35:13.550 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:35:17.525 INFO:teuthology.orchestra.run.smithi040.stdout:(1/2): ceph-debuginfo-17.2.6-1165.g93dce5e1.el8 6.0 MB/s | 24 MB 00:03 2023-09-30T00:35:21.325 INFO:teuthology.orchestra.run.smithi040.stdout:(2/2): ceph-base-debuginfo-17.2.6-1165.g93dce5e 11 MB/s | 86 MB 00:07 2023-09-30T00:35:21.326 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:35:21.327 INFO:teuthology.orchestra.run.smithi040.stdout:Total 14 MB/s | 110 MB 00:07 2023-09-30T00:35:21.327 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:35:21.332 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:35:21.332 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:35:21.853 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:35:21.857 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:35:25.163 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:35:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T00:35:36.016 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 2/2 2023-09-30T00:35:36.314 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 2/2 2023-09-30T00:35:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/2 2023-09-30T00:35:36.985 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:35:36.985 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:36.985 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:35:36.985 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:36.985 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:35:36.985 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:36.985 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:35:37.132 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-common-debuginfo 2023-09-30T00:35:38.142 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:40:16 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:35:39.922 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:35:39.923 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:35:39.923 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:35:39.923 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:35:39.923 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:35:39.923 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 557 M 2023-09-30T00:35:39.923 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:39.923 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:35:39.923 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:35:39.924 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:35:39.924 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:35:39.924 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 557 M 2023-09-30T00:35:39.924 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 2.9 G 2023-09-30T00:35:39.924 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:36:04.128 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:36:04.392 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:36:04.639 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:36:05.220 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:36:05.220 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:36:05.220 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:36:05.220 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:05.220 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:36:05.220 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:36:05.402 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-09-30T00:36:06.252 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:40:44 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:36:07.834 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:36:07.834 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2023-09-30T00:36:07.834 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:36:07.834 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2023-09-30T00:36:07.834 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:36:07.834 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.7 M 2023-09-30T00:36:07.835 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:36:07.835 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:36:07.835 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2023-09-30T00:36:07.835 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:36:07.835 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:36:07.835 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 2.7 M 2023-09-30T00:36:07.835 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 13 M 2023-09-30T00:36:07.836 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:36:09.762 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-immutable-object-cache-debuginfo-17.2.6-11 1.4 MB/s | 2.7 MB 00:01 2023-09-30T00:36:09.764 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:36:09.764 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.4 MB/s | 2.7 MB 00:01 2023-09-30T00:36:09.764 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:36:09.768 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:36:09.769 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:36:09.815 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:36:09.818 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:36:10.200 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:36:10.375 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:17.2.6-1165. 1/1 2023-09-30T00:36:10.641 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:17.2.6-1165. 1/1 2023-09-30T00:36:11.229 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:17.2.6-1165. 1/1 2023-09-30T00:36:11.229 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:36:11.229 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:36:11.229 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:36:11.229 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:36:11.229 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:36:11.345 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-radosgw-debuginfo 2023-09-30T00:36:12.200 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:40:50 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:36:13.781 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:36:13.781 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:36:13.781 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:36:13.782 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:36:13.782 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:36:13.782 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 319 M 2023-09-30T00:36:13.782 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:36:13.782 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:36:13.782 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:36:13.782 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:36:13.782 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:36:13.783 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 319 M 2023-09-30T00:36:13.783 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 1.7 G 2023-09-30T00:36:13.783 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:36:34.863 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-radosgw-debuginfo-17.2.6-1165.g93dce5e1.el 15 MB/s | 319 MB 00:21 2023-09-30T00:36:34.864 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:36:34.864 INFO:teuthology.orchestra.run.smithi135.stdout:Total 15 MB/s | 319 MB 00:21 2023-09-30T00:36:34.864 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:36:34.869 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:36:34.869 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:36:36.084 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:36:36.087 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:36:46.331 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-common-debuginfo-17.2.6-1165.g93dce5e1.el8 8.4 MB/s | 557 MB 01:06 2023-09-30T00:36:46.331 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:36:46.332 INFO:teuthology.orchestra.run.smithi040.stdout:Total 8.4 MB/s | 557 MB 01:06 2023-09-30T00:36:46.332 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:36:46.336 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:36:46.336 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:36:48.642 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:36:48.645 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:37:11.454 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:37:13.508 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x 1/1 2023-09-30T00:37:13.761 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x 1/1 2023-09-30T00:37:14.352 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x 1/1 2023-09-30T00:37:14.352 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:37:14.352 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:37:14.353 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:37:14.353 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:37:14.353 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:37:14.501 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-test-debuginfo 2023-09-30T00:37:15.347 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:41:53 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:37:16.931 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:37:16.931 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:37:16.931 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:37:16.931 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:37:16.931 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:37:16.932 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 938 M 2023-09-30T00:37:16.932 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:37:16.932 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:37:16.932 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:37:16.932 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:37:16.932 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:37:16.933 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 938 M 2023-09-30T00:37:16.933 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 0 2023-09-30T00:37:16.933 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:37:55.293 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:37:55.505 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:37:55.789 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:37:56.450 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:37:56.450 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:37:56.450 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:37:56.450 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:37:56.450 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:37:56.450 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:37:57.164 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-09-30T00:37:58.142 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:42:36 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:37:59.904 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:37:59.905 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-09-30T00:37:59.905 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:37:59.905 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-09-30T00:37:59.905 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:37:59.905 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.7 M 2023-09-30T00:37:59.905 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:37:59.905 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:37:59.905 INFO:teuthology.orchestra.run.smithi040.stdout:======================================================================================== 2023-09-30T00:37:59.905 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:37:59.906 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:37:59.906 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 2.7 M 2023-09-30T00:37:59.906 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 13 M 2023-09-30T00:37:59.906 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:38:00.694 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-immutable-object-cache-debuginfo-17.2.6-11 3.4 MB/s | 2.7 MB 00:00 2023-09-30T00:38:00.695 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:38:00.695 INFO:teuthology.orchestra.run.smithi040.stdout:Total 3.4 MB/s | 2.7 MB 00:00 2023-09-30T00:38:00.695 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:38:00.700 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:38:00.700 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:38:00.751 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:38:00.754 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:38:01.151 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:38:01.338 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:17.2.6-1165. 1/1 2023-09-30T00:38:01.622 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:17.2.6-1165. 1/1 2023-09-30T00:38:02.283 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:17.2.6-1165. 1/1 2023-09-30T00:38:02.283 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:38:02.284 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:38:02.284 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:38:02.284 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:38:02.284 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:38:03.389 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-test-debuginfo-17.2.6-1165.g93dce5e1.el8.x 20 MB/s | 938 MB 00:46 2023-09-30T00:38:03.390 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:38:03.390 INFO:teuthology.orchestra.run.smithi135.stdout:Total 20 MB/s | 938 MB 00:46 2023-09-30T00:38:03.391 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:38:03.395 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:38:03.395 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:38:03.930 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw-debuginfo 2023-09-30T00:38:04.925 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:42:42 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:38:06.675 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:38:06.676 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:38:06.676 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:38:06.676 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:38:06.676 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:38:06.676 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 319 M 2023-09-30T00:38:06.677 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:38:06.677 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:38:06.677 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:38:06.677 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:38:06.677 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:38:06.677 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 319 M 2023-09-30T00:38:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 1.7 G 2023-09-30T00:38:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:38:06.895 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:38:06.898 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:38:22.453 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-radosgw-debuginfo-17.2.6-1165.g93dce5e1.el 20 MB/s | 319 MB 00:15 2023-09-30T00:38:22.454 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:38:22.454 INFO:teuthology.orchestra.run.smithi040.stdout:Total 20 MB/s | 319 MB 00:15 2023-09-30T00:38:22.455 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:38:22.459 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:38:22.459 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:38:23.776 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:38:23.779 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:39:02.806 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:39:02.979 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x 1/1 2023-09-30T00:39:03.290 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x 1/1 2023-09-30T00:39:03.954 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x 1/1 2023-09-30T00:39:03.954 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:39:03.954 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:39:03.954 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:39:03.954 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:39:03.955 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:39:04.114 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-test-debuginfo 2023-09-30T00:39:05.093 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:43:43 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:39:06.851 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:39:06.852 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:39:06.852 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:39:06.852 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:39:06.852 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:39:06.852 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 938 M 2023-09-30T00:39:06.853 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:39:06.853 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:39:06.853 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:39:06.853 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:39:06.853 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:39:06.853 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 938 M 2023-09-30T00:39:06.854 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 0 2023-09-30T00:39:06.854 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:39:45.049 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-test-debuginfo-17.2.6-1165.g93dce5e1.el8.x 25 MB/s | 938 MB 00:38 2023-09-30T00:39:45.049 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:39:45.050 INFO:teuthology.orchestra.run.smithi040.stdout:Total 25 MB/s | 938 MB 00:38 2023-09-30T00:39:45.050 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:39:45.055 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:39:45.055 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:39:48.850 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:39:48.854 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:39:51.038 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:39:51.276 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:39:51.542 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:39:52.145 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:39:52.146 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:39:52.146 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:39:52.146 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:39:52.146 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:39:52.146 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:39:53.974 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-base-debuginfo 2023-09-30T00:39:54.827 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:44:32 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:39:56.222 INFO:teuthology.orchestra.run.smithi135.stdout:Package ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:39:56.410 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:39:56.412 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-09-30T00:39:56.412 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:39:56.565 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-debuginfo 2023-09-30T00:39:57.417 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:44:35 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:39:59.002 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:39:59.002 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:39:59.003 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:39:59.003 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:39:59.003 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:39:59.003 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 31 M 2023-09-30T00:39:59.003 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:39:59.003 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:39:59.003 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:39:59.003 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:39:59.003 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:39:59.003 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 31 M 2023-09-30T00:39:59.003 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 189 M 2023-09-30T00:39:59.003 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:40:00.780 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-mgr-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 17 MB/s | 31 MB 00:01 2023-09-30T00:40:00.781 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:40:00.781 INFO:teuthology.orchestra.run.smithi135.stdout:Total 17 MB/s | 31 MB 00:01 2023-09-30T00:40:00.782 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:40:00.785 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:40:00.786 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:40:00.933 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:40:00.936 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:40:04.504 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:40:05.498 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:40:05.785 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:40:06.367 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:40:06.367 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:40:06.367 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:40:06.367 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:06.367 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:40:06.368 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:40:06.486 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mds-debuginfo 2023-09-30T00:40:07.338 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:44:45 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:40:08.926 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:40:08.926 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:40:08.926 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:40:08.926 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:40:08.926 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:40:08.927 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 58 M 2023-09-30T00:40:08.927 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:40:08.927 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:40:08.927 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:40:08.927 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:40:08.927 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:40:08.928 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 58 M 2023-09-30T00:40:08.928 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 297 M 2023-09-30T00:40:08.928 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:40:24.356 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-mds-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 3.8 MB/s | 58 MB 00:15 2023-09-30T00:40:24.356 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:40:24.356 INFO:teuthology.orchestra.run.smithi135.stdout:Total 3.8 MB/s | 58 MB 00:15 2023-09-30T00:40:24.357 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:40:24.361 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:40:24.361 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:40:24.611 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:40:24.614 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:40:30.888 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:40:31.111 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:40:31.362 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:40:31.948 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:40:31.948 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:40:31.948 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:40:31.948 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:31.948 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:40:31.948 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:40:32.059 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mon-debuginfo 2023-09-30T00:40:32.909 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:45:10 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:40:34.489 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:40:34.490 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:40:34.491 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:40:34.491 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:40:34.491 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:40:34.491 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 92 M 2023-09-30T00:40:34.491 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:40:34.491 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:40:34.491 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:40:34.491 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:40:34.491 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:40:34.491 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 92 M 2023-09-30T00:40:34.492 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 540 M 2023-09-30T00:40:34.492 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:40:42.481 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-mon-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 12 MB/s | 92 MB 00:07 2023-09-30T00:40:42.482 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:40:42.483 INFO:teuthology.orchestra.run.smithi135.stdout:Total 12 MB/s | 92 MB 00:07 2023-09-30T00:40:42.483 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:40:42.487 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:40:42.487 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:40:42.864 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:40:42.867 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:40:53.195 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:40:53.424 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:40:53.684 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:40:54.260 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:40:54.260 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:40:54.260 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:40:54.261 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:54.261 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:40:54.261 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:40:54.391 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-osd-debuginfo 2023-09-30T00:40:55.247 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:45:33 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:40:56.825 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:40:56.826 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:40:56.826 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:40:56.826 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:40:56.826 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:40:56.826 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 312 M 2023-09-30T00:40:56.827 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:40:56.827 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:40:56.827 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:40:56.827 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:40:56.827 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:40:56.827 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 312 M 2023-09-30T00:40:56.827 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 1.7 G 2023-09-30T00:40:56.827 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:41:19.164 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-osd-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 14 MB/s | 312 MB 00:22 2023-09-30T00:41:19.165 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:41:19.165 INFO:teuthology.orchestra.run.smithi135.stdout:Total 14 MB/s | 312 MB 00:22 2023-09-30T00:41:19.166 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:41:19.169 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:41:19.170 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:41:20.358 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:41:20.361 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:41:44.024 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:41:44.265 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:41:44.571 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:41:45.231 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:41:45.231 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:41:45.231 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:41:45.231 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:45.231 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:41:45.231 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:41:45.455 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-base-debuginfo 2023-09-30T00:41:46.434 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:46:24 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:41:48.006 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:41:48.217 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:41:48.218 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2023-09-30T00:41:48.218 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:41:48.334 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-debuginfo 2023-09-30T00:41:49.313 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:46:27 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:41:51.073 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:41:51.074 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:41:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:41:51.074 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:41:51.074 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:41:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 31 M 2023-09-30T00:41:51.074 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:41:51.074 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:41:51.074 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:41:51.075 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:41:51.075 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:41:51.075 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 31 M 2023-09-30T00:41:51.075 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 189 M 2023-09-30T00:41:51.075 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:41:54.447 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:41:55.224 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:41:55.500 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:41:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:41:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:41:56.093 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:41:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:56.093 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:41:56.093 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:41:56.229 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-fuse-debuginfo 2023-09-30T00:41:57.097 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:46:35 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:41:58.156 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-mgr-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 4.4 MB/s | 31 MB 00:07 2023-09-30T00:41:58.156 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:41:58.157 INFO:teuthology.orchestra.run.smithi040.stdout:Total 4.4 MB/s | 31 MB 00:07 2023-09-30T00:41:58.157 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:41:58.162 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:41:58.162 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:41:58.325 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:41:58.328 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:41:58.692 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:41:58.693 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:41:58.693 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:41:58.693 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:41:58.693 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:41:58.693 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 12 M 2023-09-30T00:41:58.693 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:41:58.693 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:41:58.693 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:41:58.694 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:41:58.694 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:41:58.694 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 12 M 2023-09-30T00:41:58.694 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 59 M 2023-09-30T00:41:58.694 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:42:02.279 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:42:02.387 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-fuse-debuginfo-17.2.6-1165.g93dce5e1.el8.x 3.3 MB/s | 12 MB 00:03 2023-09-30T00:42:02.388 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:02.388 INFO:teuthology.orchestra.run.smithi135.stdout:Total 3.3 MB/s | 12 MB 00:03 2023-09-30T00:42:02.388 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:42:02.392 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:42:02.393 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:42:02.473 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:42:02.477 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:42:02.549 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:42:03.418 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:42:04.562 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:42:04.747 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:42:04.773 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:42:04.773 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:42:04.773 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:42:04.773 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:04.773 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:42:04.773 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:42:04.995 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:42:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:42:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:05.574 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:42:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:05.574 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:05.574 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:42:05.685 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados-devel-debuginfo 2023-09-30T00:42:06.470 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mds-debuginfo 2023-09-30T00:42:06.549 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:46:44 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:42:07.453 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:46:45 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:42:08.139 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:42:08.139 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:42:08.139 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:42:08.140 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:42:08.140 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:42:08.140 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 128 k 2023-09-30T00:42:08.140 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:08.140 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:42:08.140 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:42:08.140 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:42:08.140 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:08.140 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 128 k 2023-09-30T00:42:08.140 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 568 k 2023-09-30T00:42:08.140 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:42:08.674 INFO:teuthology.orchestra.run.smithi135.stdout:librados-devel-debuginfo-17.2.6-1165.g93dce5e1. 240 kB/s | 128 kB 00:00 2023-09-30T00:42:08.675 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:08.675 INFO:teuthology.orchestra.run.smithi135.stdout:Total 239 kB/s | 128 kB 00:00 2023-09-30T00:42:08.675 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:42:08.679 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:42:08.679 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:42:08.713 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:42:08.716 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:42:08.816 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:42:08.979 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T00:42:09.229 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:42:09.229 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:42:09.229 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:42:09.230 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:42:09.230 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:42:09.230 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 58 M 2023-09-30T00:42:09.230 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:42:09.230 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:42:09.230 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:42:09.230 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:42:09.230 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:42:09.231 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 58 M 2023-09-30T00:42:09.231 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 297 M 2023-09-30T00:42:09.231 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:42:09.245 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T00:42:10.768 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T00:42:10.768 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:10.768 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:42:10.768 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:10.768 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:10.768 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:42:12.750 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install libcephfs2-debuginfo 2023-09-30T00:42:13.627 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:46:51 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:42:15.210 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:42:15.211 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:42:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:42:15.211 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:42:15.211 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:42:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 11 M 2023-09-30T00:42:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:15.211 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:42:15.211 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:42:15.211 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:42:15.211 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:15.212 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 11 M 2023-09-30T00:42:15.212 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 55 M 2023-09-30T00:42:15.212 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:42:18.218 INFO:teuthology.orchestra.run.smithi135.stdout:libcephfs2-debuginfo-17.2.6-1165.g93dce5e1.el8. 3.8 MB/s | 11 MB 00:03 2023-09-30T00:42:18.219 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:18.219 INFO:teuthology.orchestra.run.smithi135.stdout:Total 3.8 MB/s | 11 MB 00:03 2023-09-30T00:42:18.219 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:42:18.223 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:42:18.224 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:42:18.302 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:42:18.305 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:42:19.571 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:42:19.770 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T00:42:20.023 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T00:42:20.206 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-mds-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 5.3 MB/s | 58 MB 00:10 2023-09-30T00:42:20.207 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:20.208 INFO:teuthology.orchestra.run.smithi040.stdout:Total 5.3 MB/s | 58 MB 00:10 2023-09-30T00:42:20.208 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:42:20.213 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:42:20.213 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:42:20.486 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:42:20.489 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:42:20.602 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T00:42:20.603 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:20.603 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:42:20.603 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:20.603 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:20.603 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:42:20.711 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados2-debuginfo 2023-09-30T00:42:21.566 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:46:59 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:42:23.150 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:42:23.151 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:42:23.151 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:42:23.151 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:42:23.151 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:42:23.151 INFO:teuthology.orchestra.run.smithi135.stdout: librados2-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 63 M 2023-09-30T00:42:23.151 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:23.151 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:42:23.151 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:42:23.151 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:42:23.151 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:23.152 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 63 M 2023-09-30T00:42:23.152 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 419 M 2023-09-30T00:42:23.152 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:42:27.383 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:42:27.584 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:42:27.891 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:42:28.567 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:42:28.567 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:42:28.567 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:42:28.567 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:28.567 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:42:28.567 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:42:28.710 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mon-debuginfo 2023-09-30T00:42:29.680 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:47:07 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:42:30.210 INFO:teuthology.orchestra.run.smithi135.stdout:librados2-debuginfo-17.2.6-1165.g93dce5e1.el8.x 8.9 MB/s | 63 MB 00:07 2023-09-30T00:42:30.211 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:30.212 INFO:teuthology.orchestra.run.smithi135.stdout:Total 8.9 MB/s | 63 MB 00:07 2023-09-30T00:42:30.212 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:42:30.216 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:42:30.216 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:42:30.509 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:42:30.511 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:42:31.477 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:42:31.477 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:42:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:42:31.478 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:42:31.478 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:42:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 92 M 2023-09-30T00:42:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:42:31.478 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:42:31.478 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:42:31.478 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:42:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:42:31.479 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 92 M 2023-09-30T00:42:31.479 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 540 M 2023-09-30T00:42:31.479 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:42:37.987 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:42:38.167 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:42:38.413 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:42:38.993 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:42:38.994 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:38.994 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:42:38.994 INFO:teuthology.orchestra.run.smithi135.stdout: librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:38.994 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:38.995 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:42:39.107 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librbd1-debuginfo 2023-09-30T00:42:39.116 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-mon-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 12 MB/s | 92 MB 00:07 2023-09-30T00:42:39.117 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:39.117 INFO:teuthology.orchestra.run.smithi040.stdout:Total 12 MB/s | 92 MB 00:07 2023-09-30T00:42:39.118 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:42:39.122 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:42:39.122 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:42:39.530 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:42:39.533 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:42:39.965 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:47:17 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:42:41.551 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:42:41.551 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:42:41.551 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:42:41.552 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:42:41.552 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:42:41.552 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 81 M 2023-09-30T00:42:41.552 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:41.552 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:42:41.552 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:42:41.552 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:42:41.552 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:41.553 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 81 M 2023-09-30T00:42:41.553 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 525 M 2023-09-30T00:42:41.553 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:42:48.861 INFO:teuthology.orchestra.run.smithi135.stdout:librbd1-debuginfo-17.2.6-1165.g93dce5e1.el8.x86 11 MB/s | 81 MB 00:07 2023-09-30T00:42:48.862 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:48.862 INFO:teuthology.orchestra.run.smithi135.stdout:Total 11 MB/s | 81 MB 00:07 2023-09-30T00:42:48.863 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:42:48.867 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:42:48.867 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:42:49.230 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:42:49.230 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:42:50.993 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:42:51.172 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:42:51.467 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:42:52.125 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:42:52.126 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:42:52.126 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:42:52.126 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:52.126 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:42:52.126 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:42:52.266 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-osd-debuginfo 2023-09-30T00:42:53.257 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:47:31 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:42:55.032 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:42:55.032 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:42:55.032 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:42:55.033 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:42:55.033 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:42:55.033 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 312 M 2023-09-30T00:42:55.033 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:42:55.033 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:42:55.033 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:42:55.033 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:42:55.033 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:42:55.034 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 312 M 2023-09-30T00:42:55.034 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 1.7 G 2023-09-30T00:42:55.034 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:42:58.771 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:42:59.005 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:42:59.258 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:42:59.849 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:42:59.849 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:59.849 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:42:59.849 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:42:59.849 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:42:59.850 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:42:59.961 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-cephfs-debuginfo 2023-09-30T00:43:00.814 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:47:38 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:43:02.405 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:43:02.406 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:02.406 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:43:02.406 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:02.406 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:43:02.406 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 620 k 2023-09-30T00:43:02.407 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:02.407 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:43:02.407 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:02.407 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:43:02.407 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:02.407 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 620 k 2023-09-30T00:43:02.407 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 3.0 M 2023-09-30T00:43:02.407 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:43:03.387 INFO:teuthology.orchestra.run.smithi135.stdout:python3-cephfs-debuginfo-17.2.6-1165.g93dce5e1. 633 kB/s | 620 kB 00:00 2023-09-30T00:43:03.388 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:03.388 INFO:teuthology.orchestra.run.smithi135.stdout:Total 631 kB/s | 620 kB 00:00 2023-09-30T00:43:03.389 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:43:03.393 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:43:03.393 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:43:03.433 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:43:03.436 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:43:04.143 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:43:05.721 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T00:43:05.993 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T00:43:06.572 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T00:43:06.573 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:06.573 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:43:06.573 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:06.573 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:06.573 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:43:06.705 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rados-debuginfo 2023-09-30T00:43:07.570 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:47:45 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:43:09.152 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:43:09.153 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:09.153 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:43:09.153 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:09.153 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:43:09.153 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 721 k 2023-09-30T00:43:09.153 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:09.153 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:43:09.153 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:09.154 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:43:09.154 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:09.154 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 721 k 2023-09-30T00:43:09.154 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 3.3 M 2023-09-30T00:43:09.154 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:43:09.807 INFO:teuthology.orchestra.run.smithi135.stdout:python3-rados-debuginfo-17.2.6-1165.g93dce5e1.e 1.1 MB/s | 721 kB 00:00 2023-09-30T00:43:09.807 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:09.808 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.1 MB/s | 721 kB 00:00 2023-09-30T00:43:09.808 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:43:09.811 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:43:09.811 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:43:09.848 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:43:09.851 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:43:10.003 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:43:10.162 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8. 1/1 2023-09-30T00:43:10.428 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8. 1/1 2023-09-30T00:43:11.006 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8. 1/1 2023-09-30T00:43:11.006 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:11.006 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:43:11.006 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:11.006 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:11.007 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:43:11.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rbd-debuginfo 2023-09-30T00:43:11.985 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:47:49 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:43:13.561 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:43:13.562 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:13.562 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:43:13.562 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:13.562 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:43:13.562 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.2 M 2023-09-30T00:43:13.562 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:13.562 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:43:13.562 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:13.562 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:43:13.563 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:13.563 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 1.2 M 2023-09-30T00:43:13.563 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 6.2 M 2023-09-30T00:43:13.563 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:43:13.826 INFO:teuthology.orchestra.run.smithi135.stdout:python3-rbd-debuginfo-17.2.6-1165.g93dce5e1.el8 4.5 MB/s | 1.2 MB 00:00 2023-09-30T00:43:13.826 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:13.826 INFO:teuthology.orchestra.run.smithi135.stdout:Total 4.4 MB/s | 1.2 MB 00:00 2023-09-30T00:43:13.827 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:43:13.830 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:43:13.830 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:43:13.867 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:43:13.870 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:43:14.073 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:43:14.278 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:43:14.538 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:43:15.120 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:43:15.120 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:15.121 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:43:15.121 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:15.121 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:15.121 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:43:15.229 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rgw-debuginfo 2023-09-30T00:43:15.706 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-osd-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 15 MB/s | 312 MB 00:20 2023-09-30T00:43:15.707 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:15.708 INFO:teuthology.orchestra.run.smithi040.stdout:Total 15 MB/s | 312 MB 00:20 2023-09-30T00:43:15.708 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:43:15.712 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:43:15.713 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:43:16.077 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:47:54 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:43:17.003 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:43:17.006 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:43:17.651 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:43:17.652 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:17.652 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:43:17.652 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:17.652 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:43:17.652 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 318 k 2023-09-30T00:43:17.652 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:17.653 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:43:17.653 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:17.653 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:43:17.653 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:17.653 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 318 k 2023-09-30T00:43:17.653 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 1.3 M 2023-09-30T00:43:17.653 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:43:18.111 INFO:teuthology.orchestra.run.smithi135.stdout:python3-rgw-debuginfo-17.2.6-1165.g93dce5e1.el8 695 kB/s | 318 kB 00:00 2023-09-30T00:43:18.112 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:18.112 INFO:teuthology.orchestra.run.smithi135.stdout:Total 693 kB/s | 318 kB 00:00 2023-09-30T00:43:18.112 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:43:18.116 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:43:18.116 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:43:18.150 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:43:18.153 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:43:18.261 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:43:18.461 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:43:18.730 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:43:19.327 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:43:19.327 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:19.327 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:43:19.327 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:19.327 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:19.327 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:43:19.433 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-fuse-debuginfo 2023-09-30T00:43:20.290 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:47:58 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:43:21.876 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:43:21.877 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:21.877 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:43:21.878 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:21.878 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:43:21.878 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.2 M 2023-09-30T00:43:21.878 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:21.878 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:43:21.879 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:21.879 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:43:21.879 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:21.879 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 1.2 M 2023-09-30T00:43:21.879 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 5.6 M 2023-09-30T00:43:21.880 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:43:22.405 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-fuse-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 2.3 MB/s | 1.2 MB 00:00 2023-09-30T00:43:22.406 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:22.406 INFO:teuthology.orchestra.run.smithi135.stdout:Total 2.3 MB/s | 1.2 MB 00:00 2023-09-30T00:43:22.407 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:43:22.411 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:43:22.411 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:43:22.453 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:43:22.456 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:43:23.276 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:43:25.610 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:43:26.048 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:26.648 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:43:26.761 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-mirror-debuginfo 2023-09-30T00:43:27.609 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:48:05 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:43:29.189 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:43:29.190 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:29.190 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:43:29.190 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:29.190 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:43:29.190 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 72 M 2023-09-30T00:43:29.190 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:29.190 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:43:29.191 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:29.191 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:43:29.191 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:29.191 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 72 M 2023-09-30T00:43:29.191 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 480 M 2023-09-30T00:43:29.191 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:43:32.458 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-mirror-debuginfo-17.2.6-1165.g93dce5e1.el8. 22 MB/s | 72 MB 00:03 2023-09-30T00:43:32.459 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:32.459 INFO:teuthology.orchestra.run.smithi135.stdout:Total 22 MB/s | 72 MB 00:03 2023-09-30T00:43:32.459 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:43:32.463 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:43:32.463 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:43:32.762 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:43:32.765 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:43:41.304 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:43:41.591 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T00:43:41.854 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T00:43:42.441 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T00:43:42.442 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:42.442 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:43:42.442 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:42.442 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:42.442 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:43:42.558 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-nbd-debuginfo 2023-09-30T00:43:43.413 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:48:21 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:43:44.973 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-09-30T00:43:44.974 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:44.974 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-09-30T00:43:44.974 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:44.974 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-09-30T00:43:44.974 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.0 M 2023-09-30T00:43:44.974 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:44.974 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-09-30T00:43:44.974 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-09-30T00:43:44.974 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-09-30T00:43:44.974 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:44.975 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 2.0 M 2023-09-30T00:43:44.975 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 9.8 M 2023-09-30T00:43:44.975 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-09-30T00:43:45.716 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-nbd-debuginfo-17.2.6-1165.g93dce5e1.el8.x86 2.7 MB/s | 2.0 MB 00:00 2023-09-30T00:43:45.717 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:45.717 INFO:teuthology.orchestra.run.smithi135.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2023-09-30T00:43:45.717 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-09-30T00:43:45.720 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-09-30T00:43:45.721 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-09-30T00:43:45.761 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-09-30T00:43:45.764 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-09-30T00:43:46.052 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-09-30T00:43:46.240 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:46.508 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:47.085 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:47.085 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:47.085 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-09-30T00:43:47.085 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:47.085 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:43:47.086 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-09-30T00:43:47.197 DEBUG:teuthology.parallel:result is None 2023-09-30T00:43:54.929 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:43:55.114 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:43:55.398 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:43:56.062 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:43:56.062 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:43:56.062 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:43:56.062 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:56.062 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:43:56.063 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:43:56.234 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-fuse-debuginfo 2023-09-30T00:43:57.222 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:48:35 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:43:58.982 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:43:58.983 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:43:58.984 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:43:58.984 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:43:58.984 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:43:58.984 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 12 M 2023-09-30T00:43:58.984 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:43:58.984 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:43:58.984 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:43:58.985 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:43:58.985 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:43:58.985 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 12 M 2023-09-30T00:43:58.985 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 59 M 2023-09-30T00:43:58.985 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:44:00.038 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-fuse-debuginfo-17.2.6-1165.g93dce5e1.el8.x 11 MB/s | 12 MB 00:01 2023-09-30T00:44:00.039 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:00.039 INFO:teuthology.orchestra.run.smithi040.stdout:Total 11 MB/s | 12 MB 00:01 2023-09-30T00:44:00.040 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:44:00.044 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:44:00.044 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:44:00.132 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:44:00.135 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:44:02.222 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:44:02.529 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:44:02.830 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:44:03.483 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:44:03.483 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:03.483 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:44:03.483 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:44:03.483 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:03.483 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:44:03.614 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados-devel-debuginfo 2023-09-30T00:44:04.603 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:48:42 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:44:06.368 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:44:06.369 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:44:06.369 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:06.369 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:44:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 128 k 2023-09-30T00:44:06.369 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:06.370 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:44:06.370 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:06.370 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:44:06.370 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:06.370 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 128 k 2023-09-30T00:44:06.370 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 568 k 2023-09-30T00:44:06.371 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:44:06.673 INFO:teuthology.orchestra.run.smithi040.stdout:librados-devel-debuginfo-17.2.6-1165.g93dce5e1. 424 kB/s | 128 kB 00:00 2023-09-30T00:44:06.673 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:06.674 INFO:teuthology.orchestra.run.smithi040.stdout:Total 422 kB/s | 128 kB 00:00 2023-09-30T00:44:06.674 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:44:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:44:06.678 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:44:06.720 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:44:06.723 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:44:06.823 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:44:07.003 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T00:44:07.306 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T00:44:08.426 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T00:44:08.427 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:08.427 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:44:08.427 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:44:08.427 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:08.427 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:44:10.306 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs2-debuginfo 2023-09-30T00:44:11.284 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:48:49 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:44:13.037 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:44:13.038 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:13.038 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:44:13.039 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:13.039 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:44:13.039 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 11 M 2023-09-30T00:44:13.039 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:13.039 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:44:13.039 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:13.039 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:44:13.040 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:13.040 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 11 M 2023-09-30T00:44:13.040 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 55 M 2023-09-30T00:44:13.040 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:44:14.817 INFO:teuthology.orchestra.run.smithi040.stdout:libcephfs2-debuginfo-17.2.6-1165.g93dce5e1.el8. 6.4 MB/s | 11 MB 00:01 2023-09-30T00:44:14.818 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:14.818 INFO:teuthology.orchestra.run.smithi040.stdout:Total 6.4 MB/s | 11 MB 00:01 2023-09-30T00:44:14.819 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:44:14.823 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:44:14.824 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:44:14.908 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:44:14.911 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:44:16.302 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:44:16.485 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T00:44:16.794 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T00:44:17.478 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T00:44:17.478 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:17.478 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:44:17.478 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:44:17.478 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:17.478 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:44:17.602 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados2-debuginfo 2023-09-30T00:44:18.577 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:48:56 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:44:20.366 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:44:20.367 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:44:20.367 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:20.367 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:44:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 63 M 2023-09-30T00:44:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:20.367 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:44:20.367 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:20.368 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:44:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:20.368 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 63 M 2023-09-30T00:44:20.368 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 419 M 2023-09-30T00:44:20.368 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:44:24.127 INFO:teuthology.orchestra.run.smithi040.stdout:librados2-debuginfo-17.2.6-1165.g93dce5e1.el8.x 17 MB/s | 63 MB 00:03 2023-09-30T00:44:24.128 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:24.128 INFO:teuthology.orchestra.run.smithi040.stdout:Total 17 MB/s | 63 MB 00:03 2023-09-30T00:44:24.129 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:44:24.133 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:44:24.133 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:44:24.457 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:44:24.460 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:44:32.729 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:44:32.933 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:44:33.219 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:44:33.981 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T00:44:33.982 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:33.982 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:44:33.982 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:44:33.982 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:33.982 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:44:34.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librbd1-debuginfo 2023-09-30T00:44:35.117 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:49:13 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:44:36.888 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:44:36.889 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:44:36.889 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:36.889 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:44:36.889 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 81 M 2023-09-30T00:44:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:36.890 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:44:36.890 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:36.890 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:44:36.890 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:36.890 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 81 M 2023-09-30T00:44:36.891 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 525 M 2023-09-30T00:44:36.891 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:44:40.646 INFO:teuthology.orchestra.run.smithi040.stdout:librbd1-debuginfo-17.2.6-1165.g93dce5e1.el8.x86 22 MB/s | 81 MB 00:03 2023-09-30T00:44:40.647 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:40.647 INFO:teuthology.orchestra.run.smithi040.stdout:Total 22 MB/s | 81 MB 00:03 2023-09-30T00:44:40.647 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:44:40.652 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:44:40.652 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:44:41.046 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:44:41.050 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:44:51.595 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:44:51.823 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:44:52.123 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:44:52.782 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:44:52.782 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:52.782 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:44:52.782 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:44:52.782 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:52.782 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:44:52.912 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-cephfs-debuginfo 2023-09-30T00:44:53.904 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:49:31 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:44:55.673 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:44:55.673 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:55.673 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:44:55.673 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:55.674 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:44:55.674 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 620 k 2023-09-30T00:44:55.674 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:55.674 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:44:55.674 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:44:55.674 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:44:55.675 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:55.675 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 620 k 2023-09-30T00:44:55.675 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 3.0 M 2023-09-30T00:44:55.675 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:44:55.906 INFO:teuthology.orchestra.run.smithi040.stdout:python3-cephfs-debuginfo-17.2.6-1165.g93dce5e1. 2.6 MB/s | 620 kB 00:00 2023-09-30T00:44:55.906 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:44:55.907 INFO:teuthology.orchestra.run.smithi040.stdout:Total 2.6 MB/s | 620 kB 00:00 2023-09-30T00:44:55.907 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:44:55.911 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:44:55.911 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:44:55.956 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:44:55.960 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:44:56.117 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:44:58.013 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T00:44:58.867 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T00:44:59.549 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T00:44:59.550 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:59.550 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:44:59.550 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:44:59.550 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:44:59.550 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:44:59.702 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rados-debuginfo 2023-09-30T00:45:00.692 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:49:38 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:45:02.445 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:45:02.446 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:02.446 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:45:02.446 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:02.446 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:45:02.446 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 721 k 2023-09-30T00:45:02.446 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:02.446 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:45:02.447 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:02.447 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:45:02.447 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:02.447 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 721 k 2023-09-30T00:45:02.447 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 3.3 M 2023-09-30T00:45:02.447 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:45:03.817 INFO:teuthology.orchestra.run.smithi040.stdout:python3-rados-debuginfo-17.2.6-1165.g93dce5e1.e 527 kB/s | 721 kB 00:01 2023-09-30T00:45:03.817 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:45:03.818 INFO:teuthology.orchestra.run.smithi040.stdout:Total 526 kB/s | 721 kB 00:01 2023-09-30T00:45:03.818 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:45:03.822 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:45:03.822 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:45:03.867 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:45:03.871 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:45:04.237 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:45:04.412 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8. 1/1 2023-09-30T00:45:04.716 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8. 1/1 2023-09-30T00:45:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8. 1/1 2023-09-30T00:45:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:05.372 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:45:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:45:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:05.372 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:45:05.503 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rbd-debuginfo 2023-09-30T00:45:06.482 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:49:44 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:45:08.240 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:45:08.240 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:08.240 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:45:08.241 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:08.241 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:45:08.241 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.2 M 2023-09-30T00:45:08.241 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:08.241 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:45:08.241 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:08.241 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:45:08.241 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:08.242 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 1.2 M 2023-09-30T00:45:08.242 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 6.2 M 2023-09-30T00:45:08.242 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:45:08.653 INFO:teuthology.orchestra.run.smithi040.stdout:python3-rbd-debuginfo-17.2.6-1165.g93dce5e1.el8 2.9 MB/s | 1.2 MB 00:00 2023-09-30T00:45:08.653 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:45:08.654 INFO:teuthology.orchestra.run.smithi040.stdout:Total 2.8 MB/s | 1.2 MB 00:00 2023-09-30T00:45:08.654 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:45:08.658 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:45:08.659 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:45:08.705 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:45:08.709 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:45:08.935 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:45:09.136 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:45:09.418 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:45:10.081 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:45:10.082 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:10.082 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:45:10.082 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:45:10.082 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:10.082 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:45:10.210 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rgw-debuginfo 2023-09-30T00:45:11.193 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:49:49 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:45:13.006 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:45:13.007 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:13.007 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:45:13.007 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:13.007 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:45:13.008 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 318 k 2023-09-30T00:45:13.008 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:13.008 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:45:13.008 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:13.008 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:45:13.008 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:13.009 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 318 k 2023-09-30T00:45:13.009 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 1.3 M 2023-09-30T00:45:13.009 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:45:13.213 INFO:teuthology.orchestra.run.smithi040.stdout:python3-rgw-debuginfo-17.2.6-1165.g93dce5e1.el8 1.5 MB/s | 318 kB 00:00 2023-09-30T00:45:13.214 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:45:13.214 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.5 MB/s | 318 kB 00:00 2023-09-30T00:45:13.214 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:45:13.218 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:45:13.218 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:45:13.259 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:45:13.262 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:45:13.386 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:45:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:45:13.870 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:45:14.543 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T00:45:14.543 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:14.544 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:45:14.544 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:45:14.544 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:14.544 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:45:16.471 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-fuse-debuginfo 2023-09-30T00:45:17.463 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:49:55 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:45:19.225 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:45:19.225 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:19.225 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:45:19.226 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:19.226 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:45:19.226 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.2 M 2023-09-30T00:45:19.226 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:19.226 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:45:19.226 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:19.226 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:45:19.226 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:19.227 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 1.2 M 2023-09-30T00:45:19.227 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 5.6 M 2023-09-30T00:45:19.227 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:45:20.199 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-fuse-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 1.2 MB/s | 1.2 MB 00:00 2023-09-30T00:45:20.200 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:45:20.200 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.2 MB/s | 1.2 MB 00:00 2023-09-30T00:45:20.201 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:45:20.205 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:45:20.205 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:45:20.251 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:45:20.255 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:45:20.480 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:45:20.793 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:45:21.082 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:45:21.745 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T00:45:21.745 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:21.745 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:45:21.746 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:45:21.746 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:21.746 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:45:21.866 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-mirror-debuginfo 2023-09-30T00:45:22.841 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:50:00 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:45:24.617 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:45:24.617 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:24.617 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:45:24.617 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:24.617 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:45:24.617 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 72 M 2023-09-30T00:45:24.617 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:24.617 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:45:24.618 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:24.618 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:45:24.618 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:24.618 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 72 M 2023-09-30T00:45:24.618 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 480 M 2023-09-30T00:45:24.618 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:45:31.984 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-mirror-debuginfo-17.2.6-1165.g93dce5e1.el8. 9.7 MB/s | 72 MB 00:07 2023-09-30T00:45:31.985 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:45:31.986 INFO:teuthology.orchestra.run.smithi040.stdout:Total 9.7 MB/s | 72 MB 00:07 2023-09-30T00:45:31.986 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:45:31.990 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:45:31.991 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:45:32.317 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:45:32.320 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:45:41.814 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:45:42.007 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T00:45:42.307 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T00:45:42.965 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T00:45:42.965 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:42.965 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:45:42.965 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:45:42.966 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:42.966 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:45:43.098 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-nbd-debuginfo 2023-09-30T00:45:44.085 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:50:22 ago on Fri 29 Sep 2023 11:55:22 PM UTC. 2023-09-30T00:45:45.859 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2023-09-30T00:45:45.859 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:45.859 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2023-09-30T00:45:45.860 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:45.860 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2023-09-30T00:45:45.860 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.0 M 2023-09-30T00:45:45.860 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:45.860 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2023-09-30T00:45:45.860 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2023-09-30T00:45:45.860 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2023-09-30T00:45:45.860 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:45.861 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 2.0 M 2023-09-30T00:45:45.861 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 9.8 M 2023-09-30T00:45:45.861 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2023-09-30T00:45:46.219 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-nbd-debuginfo-17.2.6-1165.g93dce5e1.el8.x86 5.6 MB/s | 2.0 MB 00:00 2023-09-30T00:45:46.219 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:45:46.220 INFO:teuthology.orchestra.run.smithi040.stdout:Total 5.6 MB/s | 2.0 MB 00:00 2023-09-30T00:45:46.220 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2023-09-30T00:45:46.224 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2023-09-30T00:45:46.225 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2023-09-30T00:45:46.274 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2023-09-30T00:45:46.278 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2023-09-30T00:45:46.598 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2023-09-30T00:45:46.773 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:47.079 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:47.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:45:47.744 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:47.744 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2023-09-30T00:45:47.744 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:45:47.744 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:47.744 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2023-09-30T00:45:47.871 DEBUG:teuthology.parallel:result is None 2023-09-30T00:45:47.871 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:45:48.014 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T00:45:48.042 INFO:teuthology.orchestra.run.smithi040.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T00:45:48.043 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T00:45:48.043 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T00:45:48.044 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:45:48.261 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T00:45:48.288 INFO:teuthology.orchestra.run.smithi135.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T00:45:48.288 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T00:45:48.289 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T00:45:48.289 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T00:45:48.289 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:45:48.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:45:48.324 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:45:48.325 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:45:48.361 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T00:45:48.361 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:45:48.361 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:45:48.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:45:48.475 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:45:48.475 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:45:48.507 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:45:48.578 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T00:45:48.579 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:45:48.579 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:45:48.614 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:45:48.688 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:45:48.689 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:45:48.718 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:45:48.787 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T00:45:48.788 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:45:48.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:45:48.823 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:45:48.899 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:45:48.899 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:45:48.930 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:45:48.999 INFO:teuthology.run_tasks:Running task ceph... 2023-09-30T00:45:49.064 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-30T00:45:49.064 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /var/log/ceph 2023-09-30T00:45:49.067 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 777 /var/log/ceph 2023-09-30T00:45:49.100 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-30T00:45:49.100 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T00:45:49.143 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T00:45:49.176 INFO:tasks.ceph:Creating extra log directories... 2023-09-30T00:45:49.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T00:45:49.219 DEBUG:teuthology.orchestra.run.smithi135:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T00:45:49.263 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-30T00:45:49.263 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-30T00:45:49.263 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406586', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi040.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPvPERG99uKqCE5cUKrNqI7eYuJHBXWPBlFgCuZx0NmMh3J9k7ndvs1VBB1bF9I+97RdCkmCFnsnYhpADHvH5Gs=', 'smithi135.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPUUjOeXF2ooqKQhpZAj7iSD5ij/F0ffRzx+l37fx47VQbEGqYBo7TgFF0BSKZOzRhalCw7yOEhzZs87TAF2ueo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-30T00:45:49.264 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T00:45:49.295 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T00:45:49.316 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T00:45:49.359 DEBUG:teuthology.orchestra.run.smithi135:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T00:45:49.392 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:45:49.392 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T00:45:49.455 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T00:45:49.455 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2023-09-30T00:45:49.519 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T00:45:49.519 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:45:49.519 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 394264 Links: 1 2023-09-30T00:45:49.519 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:45:49.519 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:45:49.520 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-09-30 00:35:04.792915791 +0000 2023-09-30T00:45:49.520 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-09-29 23:35:52.404351682 +0000 2023-09-30T00:45:49.520 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-09-29 23:35:52.404351682 +0000 2023-09-30T00:45:49.520 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-09-30T00:45:49.520 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T00:45:49.593 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-09-30T00:45:49.593 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-09-30T00:45:49.593 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000253655 s, 2.0 MB/s 2023-09-30T00:45:49.594 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T00:45:49.659 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2023-09-30T00:45:49.722 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T00:45:49.722 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:45:49.723 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 389711 Links: 1 2023-09-30T00:45:49.723 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:45:49.723 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:45:49.723 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-09-30 00:35:04.792915791 +0000 2023-09-30T00:45:49.723 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-09-29 23:35:52.983344629 +0000 2023-09-30T00:45:49.723 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-09-29 23:35:52.983344629 +0000 2023-09-30T00:45:49.723 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-09-30T00:45:49.723 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T00:45:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-09-30T00:45:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-09-30T00:45:49.797 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000240534 s, 2.1 MB/s 2023-09-30T00:45:49.798 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T00:45:49.863 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2023-09-30T00:45:49.927 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T00:45:49.927 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:45:49.927 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 388873 Links: 1 2023-09-30T00:45:49.927 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:45:49.928 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:45:49.928 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-09-30 00:35:04.792915791 +0000 2023-09-30T00:45:49.928 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-09-29 23:35:53.559337613 +0000 2023-09-30T00:45:49.928 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-09-29 23:35:53.559337613 +0000 2023-09-30T00:45:49.928 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-09-30T00:45:49.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T00:45:50.001 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-09-30T00:45:50.001 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-09-30T00:45:50.002 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000221769 s, 2.3 MB/s 2023-09-30T00:45:50.003 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T00:45:50.067 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2023-09-30T00:45:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T00:45:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:45:50.131 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 6h/6d Inode: 389843 Links: 1 2023-09-30T00:45:50.131 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:45:50.131 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:45:50.131 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2023-09-30 00:35:04.792915791 +0000 2023-09-30T00:45:50.131 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2023-09-29 23:35:54.142330512 +0000 2023-09-30T00:45:50.131 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2023-09-29 23:35:54.142330512 +0000 2023-09-30T00:45:50.132 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2023-09-30T00:45:50.132 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T00:45:50.205 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2023-09-30T00:45:50.205 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2023-09-30T00:45:50.205 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000255497 s, 2.0 MB/s 2023-09-30T00:45:50.206 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T00:45:50.271 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:45:50.271 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:45:50.272 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T00:45:50.293 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T00:45:50.293 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_1 2023-09-30T00:45:50.354 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T00:45:50.354 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:45:50.354 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 6h/6d Inode: 384747 Links: 1 2023-09-30T00:45:50.354 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:45:50.355 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:45:50.355 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2023-09-30 00:33:56.237133429 +0000 2023-09-30T00:45:50.355 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2023-09-29 23:35:06.586041245 +0000 2023-09-30T00:45:50.355 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2023-09-29 23:35:06.586041245 +0000 2023-09-30T00:45:50.355 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: - 2023-09-30T00:45:50.355 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T00:45:50.425 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2023-09-30T00:45:50.425 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2023-09-30T00:45:50.425 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000326263 s, 1.6 MB/s 2023-09-30T00:45:50.426 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T00:45:50.488 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_2 2023-09-30T00:45:50.547 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T00:45:50.547 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:45:50.548 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 6h/6d Inode: 393504 Links: 1 2023-09-30T00:45:50.548 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:45:50.548 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:45:50.548 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2023-09-30 00:33:56.237133429 +0000 2023-09-30T00:45:50.548 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2023-09-29 23:35:07.073031204 +0000 2023-09-30T00:45:50.548 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2023-09-29 23:35:07.073031204 +0000 2023-09-30T00:45:50.548 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: - 2023-09-30T00:45:50.548 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T00:45:50.615 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2023-09-30T00:45:50.615 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2023-09-30T00:45:50.615 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000224959 s, 2.3 MB/s 2023-09-30T00:45:50.616 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T00:45:50.677 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_3 2023-09-30T00:45:50.735 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T00:45:50.736 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:45:50.736 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 6h/6d Inode: 396339 Links: 1 2023-09-30T00:45:50.736 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:45:50.737 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:45:50.737 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2023-09-30 00:33:56.237133429 +0000 2023-09-30T00:45:50.737 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2023-09-29 23:35:07.617019987 +0000 2023-09-30T00:45:50.737 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2023-09-29 23:35:07.617019987 +0000 2023-09-30T00:45:50.737 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: - 2023-09-30T00:45:50.737 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T00:45:50.803 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2023-09-30T00:45:50.803 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2023-09-30T00:45:50.803 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000219965 s, 2.3 MB/s 2023-09-30T00:45:50.804 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T00:45:50.863 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_4 2023-09-30T00:45:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T00:45:50.921 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:45:50.921 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 6h/6d Inode: 389963 Links: 1 2023-09-30T00:45:50.921 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:45:50.921 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:45:50.921 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2023-09-30 00:33:56.237133429 +0000 2023-09-30T00:45:50.921 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2023-09-29 23:35:08.124009533 +0000 2023-09-30T00:45:50.922 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2023-09-29 23:35:08.124009533 +0000 2023-09-30T00:45:50.922 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: - 2023-09-30T00:45:50.922 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T00:45:50.987 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2023-09-30T00:45:50.987 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2023-09-30T00:45:50.987 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000227162 s, 2.3 MB/s 2023-09-30T00:45:50.989 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T00:45:51.048 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:45:51.048 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi135.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T00:45:51.048 INFO:tasks.ceph:Generating config... 2023-09-30T00:45:51.049 INFO:tasks.ceph:[client] debug monc = 20 2023-09-30T00:45:51.049 INFO:tasks.ceph:[client] debug ms = 1 2023-09-30T00:45:51.050 INFO:tasks.ceph:[client] debug objecter = 20 2023-09-30T00:45:51.050 INFO:tasks.ceph:[client] debug rados = 20 2023-09-30T00:45:51.050 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-30T00:45:51.050 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-09-30T00:45:51.050 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-09-30T00:45:51.050 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-09-30T00:45:51.050 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-09-30T00:45:51.050 INFO:tasks.ceph:[global] ms type = async 2023-09-30T00:45:51.050 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-09-30T00:45:51.050 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-30T00:45:51.050 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-30T00:45:51.051 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-30T00:45:51.051 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-30T00:45:51.051 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-30T00:45:51.051 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-09-30T00:45:51.051 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-09-30T00:45:51.051 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-09-30T00:45:51.051 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-09-30T00:45:51.052 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-09-30T00:45:51.052 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-30T00:45:51.052 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-09-30T00:45:51.052 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-09-30T00:45:51.052 INFO:tasks.ceph:[osd] bdev async discard = True 2023-09-30T00:45:51.052 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-09-30T00:45:51.052 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-09-30T00:45:51.052 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-30T00:45:51.052 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-30T00:45:51.052 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-30T00:45:51.052 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-09-30T00:45:51.052 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-09-30T00:45:51.053 INFO:tasks.ceph:[osd] debug monc = 20 2023-09-30T00:45:51.053 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-30T00:45:51.053 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-30T00:45:51.053 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-30T00:45:51.053 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-30T00:45:51.053 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-30T00:45:51.053 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-30T00:45:51.053 INFO:tasks.ceph:[osd] osd class default list = * 2023-09-30T00:45:51.053 INFO:tasks.ceph:[osd] osd class load list = * 2023-09-30T00:45:51.053 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-09-30T00:45:51.054 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-30T00:45:51.054 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-30T00:45:51.054 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-30T00:45:51.054 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-09-30T00:45:51.054 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-09-30T00:45:51.054 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-30T00:45:51.054 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-30T00:45:51.054 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-30T00:45:51.054 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-30T00:45:51.054 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-30T00:45:51.054 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-09-30T00:45:51.055 INFO:tasks.ceph:Setting up mon.a... 2023-09-30T00:45:51.055 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-30T00:45:51.122 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.keyring 2023-09-30T00:45:51.125 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-30T00:45:51.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:45:51.207 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.40'), ('mon.c', '[v2:172.21.15.40:3301,v1:172.21.15.40:6790]'), ('mon.b', '172.21.15.135')] 2023-09-30T00:45:51.208 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.40,[v2:172.21.15.40:3301,v1:172.21.15.40:6790],172.21.15.135', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-09-30T00:45:51.209 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:45:51.209 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T00:45:51.270 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.40 --addv c '[v2:172.21.15.40:3301,v1:172.21.15.40:6790]' --add b 172.21.15.135 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:45:51.365 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:45:51.366 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: generated fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:45:51.366 INFO:teuthology.orchestra.run.smithi040.stdout:setting min_mon_release = octopus 2023-09-30T00:45:51.366 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 0 2023-09-30T00:45:51.366 INFO:teuthology.orchestra.run.smithi040.stdout:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:45:51.366 INFO:teuthology.orchestra.run.smithi040.stdout:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:45:51.366 INFO:teuthology.orchestra.run.smithi040.stdout:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:45:51.366 INFO:teuthology.orchestra.run.smithi040.stdout:min_mon_release 15 (octopus) 2023-09-30T00:45:51.366 INFO:teuthology.orchestra.run.smithi040.stdout:election_strategy: 3 2023-09-30T00:45:51.366 INFO:teuthology.orchestra.run.smithi040.stdout:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:45:51.367 INFO:teuthology.orchestra.run.smithi040.stdout:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:45:51.367 INFO:teuthology.orchestra.run.smithi040.stdout:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:45:51.367 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-30T00:45:51.368 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T00:45:51.389 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 238dba7c-9190-48bd-bb67-322ebc9ecbd0... 2023-09-30T00:45:51.390 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T00:45:51.431 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T00:45:51.477 INFO:teuthology.orchestra.run.smithi135.stdout:[global] 2023-09-30T00:45:51.478 INFO:teuthology.orchestra.run.smithi135.stdout: chdir = "" 2023-09-30T00:45:51.478 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T00:45:51.478 INFO:teuthology.orchestra.run.smithi135.stdout: auth supported = cephx 2023-09-30T00:45:51.478 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.478 INFO:teuthology.orchestra.run.smithi135.stdout: filestore xattr use omap = true 2023-09-30T00:45:51.478 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.478 INFO:teuthology.orchestra.run.smithi135.stdout: mon clock drift allowed = 1.000 2023-09-30T00:45:51.479 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.479 INFO:teuthology.orchestra.run.smithi135.stdout: osd crush chooseleaf type = 0 2023-09-30T00:45:51.479 INFO:teuthology.orchestra.run.smithi135.stdout: auth debug = true 2023-09-30T00:45:51.479 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.479 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on old message = true 2023-09-30T00:45:51.479 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on bug = true 2023-09-30T00:45:51.479 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.479 INFO:teuthology.orchestra.run.smithi135.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T00:45:51.479 INFO:teuthology.orchestra.run.smithi135.stdout: mon pg warn max object skew = 0 2023-09-30T00:45:51.479 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.480 INFO:teuthology.orchestra.run.smithi135.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T00:45:51.480 INFO:teuthology.orchestra.run.smithi135.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T00:45:51.480 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.480 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default size = 2 2023-09-30T00:45:51.480 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.480 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow primary affinity = true 2023-09-30T00:45:51.480 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow pg remap = true 2023-09-30T00:45:51.480 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on legacy crush tunables = false 2023-09-30T00:45:51.480 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on crush straw calc version zero = false 2023-09-30T00:45:51.480 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on no sortbitwise = false 2023-09-30T00:45:51.481 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on osd down out interval zero = false 2023-09-30T00:45:51.481 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on too few osds = false 2023-09-30T00:45:51.481 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T00:45:51.481 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T00:45:51.481 INFO:teuthology.orchestra.run.smithi135.stdout: mon_allow_pool_size_one = true 2023-09-30T00:45:51.481 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.481 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T00:45:51.481 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.481 INFO:teuthology.orchestra.run.smithi135.stdout: osd default data pool replay window = 5 2023-09-30T00:45:51.481 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.482 INFO:teuthology.orchestra.run.smithi135.stdout: mon allow pool delete = true 2023-09-30T00:45:51.482 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.482 INFO:teuthology.orchestra.run.smithi135.stdout: mon cluster log file level = debug 2023-09-30T00:45:51.482 INFO:teuthology.orchestra.run.smithi135.stdout: debug asserts on shutdown = true 2023-09-30T00:45:51.482 INFO:teuthology.orchestra.run.smithi135.stdout: mon health detail to clog = false 2023-09-30T00:45:51.482 INFO:teuthology.orchestra.run.smithi135.stdout: mon host = "172.21.15.40,[v2:172.21.15.40:3301,v1:172.21.15.40:6790],172.21.15.135" 2023-09-30T00:45:51.482 INFO:teuthology.orchestra.run.smithi135.stdout: mon client directed command retry = 5 2023-09-30T00:45:51.482 INFO:teuthology.orchestra.run.smithi135.stdout: mon election default strategy = 3 2023-09-30T00:45:51.482 INFO:teuthology.orchestra.run.smithi135.stdout: ms bind msgr1 = False 2023-09-30T00:45:51.483 INFO:teuthology.orchestra.run.smithi135.stdout: ms bind msgr2 = True 2023-09-30T00:45:51.483 INFO:teuthology.orchestra.run.smithi135.stdout: ms inject socket failures = 5000 2023-09-30T00:45:51.483 INFO:teuthology.orchestra.run.smithi135.stdout: ms type = async 2023-09-30T00:45:51.483 INFO:teuthology.orchestra.run.smithi135.stdout: osd heartbeat grace = 80 2023-09-30T00:45:51.483 INFO:teuthology.orchestra.run.smithi135.stdout: fsid = 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:45:51.483 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.483 INFO:teuthology.orchestra.run.smithi135.stdout:[osd] 2023-09-30T00:45:51.483 INFO:teuthology.orchestra.run.smithi135.stdout: osd journal size = 100 2023-09-30T00:45:51.483 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.483 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub load threshold = 5.0 2023-09-30T00:45:51.484 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub max interval = 600 2023-09-30T00:45:51.484 INFO:teuthology.orchestra.run.smithi135.stdout: osd mclock profile = high_recovery_ops 2023-09-30T00:45:51.484 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.484 INFO:teuthology.orchestra.run.smithi135.stdout: osd recover clone overlap = true 2023-09-30T00:45:51.484 INFO:teuthology.orchestra.run.smithi135.stdout: osd recovery max chunk = 1048576 2023-09-30T00:45:51.484 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.484 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug shutdown = true 2023-09-30T00:45:51.484 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug op order = true 2023-09-30T00:45:51.484 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify stray on activate = true 2023-09-30T00:45:51.485 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.485 INFO:teuthology.orchestra.run.smithi135.stdout: osd open classes on start = true 2023-09-30T00:45:51.485 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug pg log writeout = true 2023-09-30T00:45:51.485 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.485 INFO:teuthology.orchestra.run.smithi135.stdout: osd deep scrub update digest min age = 30 2023-09-30T00:45:51.485 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.485 INFO:teuthology.orchestra.run.smithi135.stdout: osd map max advance = 10 2023-09-30T00:45:51.485 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.485 INFO:teuthology.orchestra.run.smithi135.stdout: journal zero on create = true 2023-09-30T00:45:51.485 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.485 INFO:teuthology.orchestra.run.smithi135.stdout: filestore ondisk finisher threads = 3 2023-09-30T00:45:51.486 INFO:teuthology.orchestra.run.smithi135.stdout: filestore apply finisher threads = 3 2023-09-30T00:45:51.486 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.486 INFO:teuthology.orchestra.run.smithi135.stdout: bdev debug aio = true 2023-09-30T00:45:51.486 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug misdirected ops = true 2023-09-30T00:45:51.486 INFO:teuthology.orchestra.run.smithi135.stdout: bdev async discard = True 2023-09-30T00:45:51.486 INFO:teuthology.orchestra.run.smithi135.stdout: bdev enable discard = True 2023-09-30T00:45:51.486 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore allocator = bitmap 2023-09-30T00:45:51.486 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore block size = 96636764160 2023-09-30T00:45:51.486 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore fsck on mount = True 2023-09-30T00:45:51.486 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore zero block detection = True 2023-09-30T00:45:51.487 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluefs = 1 2023-09-30T00:45:51.487 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluestore = 1 2023-09-30T00:45:51.487 INFO:teuthology.orchestra.run.smithi135.stdout: debug monc = 20 2023-09-30T00:45:51.487 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2023-09-30T00:45:51.487 INFO:teuthology.orchestra.run.smithi135.stdout: debug osd = 20 2023-09-30T00:45:51.487 INFO:teuthology.orchestra.run.smithi135.stdout: debug rocksdb = 10 2023-09-30T00:45:51.487 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T00:45:51.487 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd full ratio = 0.9 2023-09-30T00:45:51.487 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd nearfull ratio = 0.8 2023-09-30T00:45:51.487 INFO:teuthology.orchestra.run.smithi135.stdout: osd class default list = * 2023-09-30T00:45:51.488 INFO:teuthology.orchestra.run.smithi135.stdout: osd class load list = * 2023-09-30T00:45:51.488 INFO:teuthology.orchestra.run.smithi135.stdout: osd client watch timeout = 120 2023-09-30T00:45:51.488 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify cached snaps = True 2023-09-30T00:45:51.488 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify missing on start = True 2023-09-30T00:45:51.488 INFO:teuthology.orchestra.run.smithi135.stdout: osd failsafe full ratio = 0.95 2023-09-30T00:45:51.488 INFO:teuthology.orchestra.run.smithi135.stdout: osd fast shutdown = False 2023-09-30T00:45:51.488 INFO:teuthology.orchestra.run.smithi135.stdout: osd max markdown count = 1000 2023-09-30T00:45:51.488 INFO:teuthology.orchestra.run.smithi135.stdout: osd mclock override recovery settings = True 2023-09-30T00:45:51.488 INFO:teuthology.orchestra.run.smithi135.stdout: osd objectstore = bluestore 2023-09-30T00:45:51.488 INFO:teuthology.orchestra.run.smithi135.stdout: osd op queue = debug_random 2023-09-30T00:45:51.489 INFO:teuthology.orchestra.run.smithi135.stdout: osd op queue cut off = debug_random 2023-09-30T00:45:51.489 INFO:teuthology.orchestra.run.smithi135.stdout: osd shutdown pgref assert = True 2023-09-30T00:45:51.489 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.489 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr] 2023-09-30T00:45:51.489 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2023-09-30T00:45:51.489 INFO:teuthology.orchestra.run.smithi135.stdout: debug mgr = 20 2023-09-30T00:45:51.489 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2023-09-30T00:45:51.489 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2023-09-30T00:45:51.489 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:45:51.489 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:45:51.490 INFO:teuthology.orchestra.run.smithi135.stdout: mgr/telemetry/nag = false 2023-09-30T00:45:51.490 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.490 INFO:teuthology.orchestra.run.smithi135.stdout:[mon] 2023-09-30T00:45:51.490 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2023-09-30T00:45:51.490 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2023-09-30T00:45:51.490 INFO:teuthology.orchestra.run.smithi135.stdout: debug paxos = 20 2023-09-30T00:45:51.490 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2023-09-30T00:45:51.490 INFO:teuthology.orchestra.run.smithi135.stdout: mon data avail warn = 5 2023-09-30T00:45:51.490 INFO:teuthology.orchestra.run.smithi135.stdout: mon mgr mkfs grace = 240 2023-09-30T00:45:51.491 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:45:51.491 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd reporter subtree level = osd 2023-09-30T00:45:51.491 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd prime pg temp = true 2023-09-30T00:45:51.491 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:45:51.491 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.491 INFO:teuthology.orchestra.run.smithi135.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T00:45:51.491 INFO:teuthology.orchestra.run.smithi135.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T00:45:51.491 INFO:teuthology.orchestra.run.smithi135.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T00:45:51.491 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.492 INFO:teuthology.orchestra.run.smithi135.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T00:45:51.492 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T00:45:51.492 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T00:45:51.492 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.492 INFO:teuthology.orchestra.run.smithi135.stdout: # 1m isn't quite enough 2023-09-30T00:45:51.492 INFO:teuthology.orchestra.run.smithi135.stdout: mon_down_mkfs_grace = 2m 2023-09-30T00:45:51.492 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.492 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_filestore_osds = false 2023-09-30T00:45:51.492 INFO:teuthology.orchestra.run.smithi135.stdout: mon min osdmap epochs = 50 2023-09-30T00:45:51.492 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd crush smoke test = False 2023-09-30T00:45:51.493 INFO:teuthology.orchestra.run.smithi135.stdout: mon osdmap full prune interval = 2 2023-09-30T00:45:51.493 INFO:teuthology.orchestra.run.smithi135.stdout: mon osdmap full prune min = 15 2023-09-30T00:45:51.493 INFO:teuthology.orchestra.run.smithi135.stdout: mon osdmap full prune txsize = 2 2023-09-30T00:45:51.493 INFO:teuthology.orchestra.run.smithi135.stdout: mon scrub interval = 300 2023-09-30T00:45:51.493 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on pool no app = False 2023-09-30T00:45:51.493 INFO:teuthology.orchestra.run.smithi135.stdout: paxos service trim min = 10 2023-09-30T00:45:51.493 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-09-30T00:45:51.493 INFO:teuthology.orchestra.run.smithi135.stdout:[client] 2023-09-30T00:45:51.493 INFO:teuthology.orchestra.run.smithi135.stdout: rgw cache enabled = true 2023-09-30T00:45:51.493 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable ops log = true 2023-09-30T00:45:51.494 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable usage log = true 2023-09-30T00:45:51.494 INFO:teuthology.orchestra.run.smithi135.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T00:45:51.494 INFO:teuthology.orchestra.run.smithi135.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T00:45:51.494 INFO:teuthology.orchestra.run.smithi135.stdout: debug monc = 20 2023-09-30T00:45:51.494 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2023-09-30T00:45:51.494 INFO:teuthology.orchestra.run.smithi135.stdout: debug objecter = 20 2023-09-30T00:45:51.494 INFO:teuthology.orchestra.run.smithi135.stdout: debug rados = 20 2023-09-30T00:45:51.494 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.a] 2023-09-30T00:45:51.494 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.c] 2023-09-30T00:45:51.494 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.b] 2023-09-30T00:45:51.495 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2023-09-30T00:45:51.495 INFO:teuthology.orchestra.run.smithi040.stdout: chdir = "" 2023-09-30T00:45:51.495 INFO:teuthology.orchestra.run.smithi040.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T00:45:51.495 INFO:teuthology.orchestra.run.smithi040.stdout: auth supported = cephx 2023-09-30T00:45:51.496 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.496 INFO:teuthology.orchestra.run.smithi040.stdout: filestore xattr use omap = true 2023-09-30T00:45:51.496 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.496 INFO:teuthology.orchestra.run.smithi040.stdout: mon clock drift allowed = 1.000 2023-09-30T00:45:51.496 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.496 INFO:teuthology.orchestra.run.smithi040.stdout: osd crush chooseleaf type = 0 2023-09-30T00:45:51.496 INFO:teuthology.orchestra.run.smithi040.stdout: auth debug = true 2023-09-30T00:45:51.496 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.496 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on old message = true 2023-09-30T00:45:51.496 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on bug = true 2023-09-30T00:45:51.497 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.497 INFO:teuthology.orchestra.run.smithi040.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T00:45:51.497 INFO:teuthology.orchestra.run.smithi040.stdout: mon pg warn max object skew = 0 2023-09-30T00:45:51.497 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.497 INFO:teuthology.orchestra.run.smithi040.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T00:45:51.497 INFO:teuthology.orchestra.run.smithi040.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T00:45:51.497 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.497 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default size = 2 2023-09-30T00:45:51.497 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.497 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow primary affinity = true 2023-09-30T00:45:51.498 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow pg remap = true 2023-09-30T00:45:51.498 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on legacy crush tunables = false 2023-09-30T00:45:51.498 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on crush straw calc version zero = false 2023-09-30T00:45:51.498 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on no sortbitwise = false 2023-09-30T00:45:51.498 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on osd down out interval zero = false 2023-09-30T00:45:51.498 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on too few osds = false 2023-09-30T00:45:51.498 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T00:45:51.498 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T00:45:51.498 INFO:teuthology.orchestra.run.smithi040.stdout: mon_allow_pool_size_one = true 2023-09-30T00:45:51.499 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.499 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T00:45:51.499 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.499 INFO:teuthology.orchestra.run.smithi040.stdout: osd default data pool replay window = 5 2023-09-30T00:45:51.499 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.499 INFO:teuthology.orchestra.run.smithi040.stdout: mon allow pool delete = true 2023-09-30T00:45:51.499 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.499 INFO:teuthology.orchestra.run.smithi040.stdout: mon cluster log file level = debug 2023-09-30T00:45:51.499 INFO:teuthology.orchestra.run.smithi040.stdout: debug asserts on shutdown = true 2023-09-30T00:45:51.499 INFO:teuthology.orchestra.run.smithi040.stdout: mon health detail to clog = false 2023-09-30T00:45:51.499 INFO:teuthology.orchestra.run.smithi040.stdout: mon host = "172.21.15.40,[v2:172.21.15.40:3301,v1:172.21.15.40:6790],172.21.15.135" 2023-09-30T00:45:51.500 INFO:teuthology.orchestra.run.smithi040.stdout: mon client directed command retry = 5 2023-09-30T00:45:51.500 INFO:teuthology.orchestra.run.smithi040.stdout: mon election default strategy = 3 2023-09-30T00:45:51.500 INFO:teuthology.orchestra.run.smithi040.stdout: ms bind msgr1 = False 2023-09-30T00:45:51.500 INFO:teuthology.orchestra.run.smithi040.stdout: ms bind msgr2 = True 2023-09-30T00:45:51.500 INFO:teuthology.orchestra.run.smithi040.stdout: ms inject socket failures = 5000 2023-09-30T00:45:51.500 INFO:teuthology.orchestra.run.smithi040.stdout: ms type = async 2023-09-30T00:45:51.500 INFO:teuthology.orchestra.run.smithi040.stdout: osd heartbeat grace = 80 2023-09-30T00:45:51.500 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:45:51.500 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.500 INFO:teuthology.orchestra.run.smithi040.stdout:[osd] 2023-09-30T00:45:51.501 INFO:teuthology.orchestra.run.smithi040.stdout: osd journal size = 100 2023-09-30T00:45:51.501 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.501 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub load threshold = 5.0 2023-09-30T00:45:51.501 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub max interval = 600 2023-09-30T00:45:51.501 INFO:teuthology.orchestra.run.smithi040.stdout: osd mclock profile = high_recovery_ops 2023-09-30T00:45:51.501 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.501 INFO:teuthology.orchestra.run.smithi040.stdout: osd recover clone overlap = true 2023-09-30T00:45:51.501 INFO:teuthology.orchestra.run.smithi040.stdout: osd recovery max chunk = 1048576 2023-09-30T00:45:51.501 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.501 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug shutdown = true 2023-09-30T00:45:51.502 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug op order = true 2023-09-30T00:45:51.502 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify stray on activate = true 2023-09-30T00:45:51.502 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.502 INFO:teuthology.orchestra.run.smithi040.stdout: osd open classes on start = true 2023-09-30T00:45:51.502 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug pg log writeout = true 2023-09-30T00:45:51.502 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.502 INFO:teuthology.orchestra.run.smithi040.stdout: osd deep scrub update digest min age = 30 2023-09-30T00:45:51.502 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.502 INFO:teuthology.orchestra.run.smithi040.stdout: osd map max advance = 10 2023-09-30T00:45:51.503 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.503 INFO:teuthology.orchestra.run.smithi040.stdout: journal zero on create = true 2023-09-30T00:45:51.503 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.503 INFO:teuthology.orchestra.run.smithi040.stdout: filestore ondisk finisher threads = 3 2023-09-30T00:45:51.503 INFO:teuthology.orchestra.run.smithi040.stdout: filestore apply finisher threads = 3 2023-09-30T00:45:51.503 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.503 INFO:teuthology.orchestra.run.smithi040.stdout: bdev debug aio = true 2023-09-30T00:45:51.503 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug misdirected ops = true 2023-09-30T00:45:51.503 INFO:teuthology.orchestra.run.smithi040.stdout: bdev async discard = True 2023-09-30T00:45:51.503 INFO:teuthology.orchestra.run.smithi040.stdout: bdev enable discard = True 2023-09-30T00:45:51.504 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore allocator = bitmap 2023-09-30T00:45:51.504 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore block size = 96636764160 2023-09-30T00:45:51.504 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore fsck on mount = True 2023-09-30T00:45:51.504 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore zero block detection = True 2023-09-30T00:45:51.504 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluefs = 1 2023-09-30T00:45:51.504 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluestore = 1 2023-09-30T00:45:51.504 INFO:teuthology.orchestra.run.smithi040.stdout: debug monc = 20 2023-09-30T00:45:51.504 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2023-09-30T00:45:51.504 INFO:teuthology.orchestra.run.smithi040.stdout: debug osd = 20 2023-09-30T00:45:51.504 INFO:teuthology.orchestra.run.smithi040.stdout: debug rocksdb = 10 2023-09-30T00:45:51.504 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T00:45:51.504 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd full ratio = 0.9 2023-09-30T00:45:51.505 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd nearfull ratio = 0.8 2023-09-30T00:45:51.505 INFO:teuthology.orchestra.run.smithi040.stdout: osd class default list = * 2023-09-30T00:45:51.505 INFO:teuthology.orchestra.run.smithi040.stdout: osd class load list = * 2023-09-30T00:45:51.505 INFO:teuthology.orchestra.run.smithi040.stdout: osd client watch timeout = 120 2023-09-30T00:45:51.505 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify cached snaps = True 2023-09-30T00:45:51.505 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify missing on start = True 2023-09-30T00:45:51.505 INFO:teuthology.orchestra.run.smithi040.stdout: osd failsafe full ratio = 0.95 2023-09-30T00:45:51.505 INFO:teuthology.orchestra.run.smithi040.stdout: osd fast shutdown = False 2023-09-30T00:45:51.505 INFO:teuthology.orchestra.run.smithi040.stdout: osd max markdown count = 1000 2023-09-30T00:45:51.505 INFO:teuthology.orchestra.run.smithi040.stdout: osd mclock override recovery settings = True 2023-09-30T00:45:51.505 INFO:teuthology.orchestra.run.smithi040.stdout: osd objectstore = bluestore 2023-09-30T00:45:51.506 INFO:teuthology.orchestra.run.smithi040.stdout: osd op queue = debug_random 2023-09-30T00:45:51.506 INFO:teuthology.orchestra.run.smithi040.stdout: osd op queue cut off = debug_random 2023-09-30T00:45:51.506 INFO:teuthology.orchestra.run.smithi040.stdout: osd shutdown pgref assert = True 2023-09-30T00:45:51.506 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.506 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr] 2023-09-30T00:45:51.506 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2023-09-30T00:45:51.506 INFO:teuthology.orchestra.run.smithi040.stdout: debug mgr = 20 2023-09-30T00:45:51.506 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2023-09-30T00:45:51.506 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2023-09-30T00:45:51.506 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:45:51.507 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:45:51.507 INFO:teuthology.orchestra.run.smithi040.stdout: mgr/telemetry/nag = false 2023-09-30T00:45:51.507 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.507 INFO:teuthology.orchestra.run.smithi040.stdout:[mon] 2023-09-30T00:45:51.507 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2023-09-30T00:45:51.507 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2023-09-30T00:45:51.507 INFO:teuthology.orchestra.run.smithi040.stdout: debug paxos = 20 2023-09-30T00:45:51.507 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2023-09-30T00:45:51.507 INFO:teuthology.orchestra.run.smithi040.stdout: mon data avail warn = 5 2023-09-30T00:45:51.507 INFO:teuthology.orchestra.run.smithi040.stdout: mon mgr mkfs grace = 240 2023-09-30T00:45:51.508 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:45:51.508 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd reporter subtree level = osd 2023-09-30T00:45:51.508 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd prime pg temp = true 2023-09-30T00:45:51.508 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:45:51.508 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.508 INFO:teuthology.orchestra.run.smithi040.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T00:45:51.508 INFO:teuthology.orchestra.run.smithi040.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T00:45:51.508 INFO:teuthology.orchestra.run.smithi040.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T00:45:51.508 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.508 INFO:teuthology.orchestra.run.smithi040.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T00:45:51.509 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T00:45:51.509 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T00:45:51.509 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.509 INFO:teuthology.orchestra.run.smithi040.stdout: # 1m isn't quite enough 2023-09-30T00:45:51.509 INFO:teuthology.orchestra.run.smithi040.stdout: mon_down_mkfs_grace = 2m 2023-09-30T00:45:51.509 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.509 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_filestore_osds = false 2023-09-30T00:45:51.509 INFO:teuthology.orchestra.run.smithi040.stdout: mon min osdmap epochs = 50 2023-09-30T00:45:51.509 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd crush smoke test = False 2023-09-30T00:45:51.509 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune interval = 2 2023-09-30T00:45:51.509 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune min = 15 2023-09-30T00:45:51.510 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune txsize = 2 2023-09-30T00:45:51.510 INFO:teuthology.orchestra.run.smithi040.stdout: mon scrub interval = 300 2023-09-30T00:45:51.510 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on pool no app = False 2023-09-30T00:45:51.510 INFO:teuthology.orchestra.run.smithi040.stdout: paxos service trim min = 10 2023-09-30T00:45:51.510 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:45:51.510 INFO:teuthology.orchestra.run.smithi040.stdout:[client] 2023-09-30T00:45:51.510 INFO:teuthology.orchestra.run.smithi040.stdout: rgw cache enabled = true 2023-09-30T00:45:51.510 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable ops log = true 2023-09-30T00:45:51.510 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable usage log = true 2023-09-30T00:45:51.510 INFO:teuthology.orchestra.run.smithi040.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T00:45:51.511 INFO:teuthology.orchestra.run.smithi040.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T00:45:51.511 INFO:teuthology.orchestra.run.smithi040.stdout: debug monc = 20 2023-09-30T00:45:51.511 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2023-09-30T00:45:51.511 INFO:teuthology.orchestra.run.smithi040.stdout: debug objecter = 20 2023-09-30T00:45:51.511 INFO:teuthology.orchestra.run.smithi040.stdout: debug rados = 20 2023-09-30T00:45:51.511 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.a] 2023-09-30T00:45:51.511 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.c] 2023-09-30T00:45:51.511 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.b] 2023-09-30T00:45:51.513 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-30T00:45:51.513 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-30T00:45:51.577 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-30T00:45:51.577 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:45:51.578 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-30T00:45:51.600 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:45:51.600 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-30T00:45:51.663 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi040.front.sepia.ceph.com 2023-09-30T00:45:51.663 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:45:51.663 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T00:45:51.663 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:45:51.750 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:45:51.751 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:45:51.813 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi135.front.sepia.ceph.com 2023-09-30T00:45:51.814 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:45:51.814 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T00:45:51.814 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:45:51.852 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:45:51.852 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:45:51.910 INFO:tasks.ceph:Setting up mon nodes... 2023-09-30T00:45:51.911 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-30T00:45:51.911 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T00:45:51.975 INFO:teuthology.orchestra.run.smithi040.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T00:45:51.978 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T00:45:52.038 INFO:teuthology.orchestra.run.smithi135.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T00:45:52.040 INFO:tasks.ceph:Setting up mds nodes... 2023-09-30T00:45:52.041 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-30T00:45:52.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-30T00:45:52.093 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-30T00:45:52.107 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-09-30T00:45:52.148 INFO:teuthology.orchestra.run.smithi135.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-09-30T00:45:52.160 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-30T00:45:52.160 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi135.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T00:45:52.160 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-30T00:45:52.195 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:45:52.195 INFO:tasks.ceph:role: osd.0 2023-09-30T00:45:52.195 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com 2023-09-30T00:45:52.196 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T00:45:52.279 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:45:52.279 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:45:52.279 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:45:52.279 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:45:52.279 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:45:52.279 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:45:52.279 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:45:52.280 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:45:52.280 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:45:52.280 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:45:52.711 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2023-09-30T00:45:52.713 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2023-09-30T00:45:52.713 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-30T00:45:52.886 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-30T00:45:52.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-30T00:45:53.005 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:45:53.005 INFO:tasks.ceph:role: osd.1 2023-09-30T00:45:53.005 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com 2023-09-30T00:45:53.005 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T00:45:53.088 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:45:53.088 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:45:53.088 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:45:53.088 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:45:53.089 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:45:53.089 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:45:53.089 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:45:53.089 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:45:53.089 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:45:53.089 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:45:53.494 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2023-09-30T00:45:53.496 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2023-09-30T00:45:53.497 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-30T00:45:53.614 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-30T00:45:53.653 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-30T00:45:53.729 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:45:53.729 INFO:tasks.ceph:role: osd.2 2023-09-30T00:45:53.729 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com 2023-09-30T00:45:53.730 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T00:45:53.813 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:45:53.813 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:45:53.813 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:45:53.813 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:45:53.814 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:45:53.814 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:45:53.814 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:45:53.814 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:45:53.814 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:45:53.814 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:45:54.261 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2023-09-30T00:45:54.263 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2023-09-30T00:45:54.263 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-30T00:45:54.443 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-30T00:45:54.483 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-30T00:45:54.559 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T00:45:54.559 INFO:tasks.ceph:role: osd.3 2023-09-30T00:45:54.559 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi040.front.sepia.ceph.com 2023-09-30T00:45:54.559 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T00:45:54.642 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:45:54.643 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:45:54.643 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:45:54.643 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:45:54.643 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:45:54.643 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:45:54.643 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:45:54.643 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:45:54.643 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:45:54.643 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:45:55.073 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2023-09-30T00:45:55.074 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2023-09-30T00:45:55.075 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-09-30T00:45:55.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-30T00:45:55.252 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:45:55.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:55.366+0000 7f93c0f71540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-30T00:45:55.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:55.366+0000 7f93c0f71540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-30T00:45:55.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:55.366+0000 7f93c0f71540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:45:55.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:55.366+0000 7f93c0f71540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:45:55.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:55.366+0000 7f93c0f71540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:45:55.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:55.367+0000 7f93c0f71540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-30T00:45:55.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:55.891+0000 7f93c0f71540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:45:56.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:56.653+0000 7f93c0f71540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:45:57.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:57.416+0000 7f93c0f71540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:45:58.204 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-30T00:45:58.240 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:45:58.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:58.334+0000 7f7f3ecbe540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-30T00:45:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:58.335+0000 7f7f3ecbe540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-30T00:45:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:58.335+0000 7f7f3ecbe540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:45:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:58.335+0000 7f7f3ecbe540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:45:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:58.335+0000 7f7f3ecbe540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:45:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:58.335+0000 7f7f3ecbe540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-30T00:45:58.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:58.862+0000 7f7f3ecbe540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:45:59.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:45:59.625+0000 7f7f3ecbe540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:00.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:00.388+0000 7f7f3ecbe540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:01.184 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-30T00:46:01.219 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:01.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:01.314+0000 7fea85af2540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-30T00:46:01.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:01.314+0000 7fea85af2540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-30T00:46:01.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:01.314+0000 7fea85af2540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:46:01.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:01.314+0000 7fea85af2540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:46:01.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:01.314+0000 7fea85af2540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:46:01.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:01.315+0000 7fea85af2540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-30T00:46:01.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:01.843+0000 7fea85af2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:02.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:02.604+0000 7fea85af2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:03.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:03.368+0000 7fea85af2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:04.162 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-30T00:46:04.197 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:04.292+0000 7fb326226540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-30T00:46:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:04.293+0000 7fb326226540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-30T00:46:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:04.293+0000 7fb326226540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:46:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:04.293+0000 7fb326226540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:46:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:04.293+0000 7fb326226540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:46:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:04.293+0000 7fb326226540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-09-30T00:46:04.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:04.818+0000 7fb326226540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:05.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:05.591+0000 7fb326226540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:06.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:06.353+0000 7fb326226540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:07.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-30T00:46:07.177 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-09-30T00:46:07.210 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:46:07.210 INFO:tasks.ceph:role: osd.4 2023-09-30T00:46:07.210 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi135.front.sepia.ceph.com 2023-09-30T00:46:07.210 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T00:46:07.288 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:46:07.288 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:46:07.288 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:46:07.288 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:46:07.289 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:46:07.289 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:46:07.289 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:46:07.289 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:46:07.289 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:46:07.289 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:46:07.714 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2023-09-30T00:46:07.716 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2023-09-30T00:46:07.716 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-09-30T00:46:07.880 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-09-30T00:46:07.917 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-09-30T00:46:07.987 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:46:07.988 INFO:tasks.ceph:role: osd.5 2023-09-30T00:46:07.988 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi135.front.sepia.ceph.com 2023-09-30T00:46:07.988 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T00:46:08.064 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:46:08.065 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:46:08.065 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:46:08.065 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:46:08.065 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:46:08.065 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:46:08.065 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:46:08.065 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:46:08.065 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:46:08.065 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:46:08.486 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2023-09-30T00:46:08.488 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2023-09-30T00:46:08.488 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-09-30T00:46:08.604 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-09-30T00:46:08.640 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-09-30T00:46:08.711 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:46:08.711 INFO:tasks.ceph:role: osd.6 2023-09-30T00:46:08.711 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi135.front.sepia.ceph.com 2023-09-30T00:46:08.711 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T00:46:08.789 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:46:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:46:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:46:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:46:08.789 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:46:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:46:08.789 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:46:08.789 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:46:08.789 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:46:08.790 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:46:09.232 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2023-09-30T00:46:09.234 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2023-09-30T00:46:09.235 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-09-30T00:46:09.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-09-30T00:46:09.441 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-09-30T00:46:09.512 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T00:46:09.512 INFO:tasks.ceph:role: osd.7 2023-09-30T00:46:09.512 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi135.front.sepia.ceph.com 2023-09-30T00:46:09.513 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T00:46:09.589 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:46:09.590 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:46:09.590 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:46:09.590 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:46:09.590 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:46:09.590 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:46:09.590 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:46:09.590 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:46:09.590 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:46:09.590 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:46:10.032 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2023-09-30T00:46:10.034 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2023-09-30T00:46:10.035 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-09-30T00:46:10.174 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-09-30T00:46:10.210 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:10.303 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:10.302+0000 7f742b8c8540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-09-30T00:46:10.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:10.302+0000 7f742b8c8540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-09-30T00:46:10.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:10.302+0000 7f742b8c8540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:46:10.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:10.302+0000 7f742b8c8540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:46:10.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:10.302+0000 7f742b8c8540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:46:10.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:10.302+0000 7f742b8c8540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-09-30T00:46:10.827 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:10.826+0000 7f742b8c8540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:11.596 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:11.595+0000 7f742b8c8540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:12.358 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:12.357+0000 7f742b8c8540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:13.144 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-09-30T00:46:13.178 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:13.268 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:13.266+0000 7f22a10ed540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-09-30T00:46:13.268 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:13.266+0000 7f22a10ed540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-09-30T00:46:13.268 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:13.266+0000 7f22a10ed540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:46:13.268 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:13.266+0000 7f22a10ed540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:46:13.268 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:13.266+0000 7f22a10ed540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:46:13.268 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:13.267+0000 7f22a10ed540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-09-30T00:46:13.795 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:13.793+0000 7f22a10ed540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:14.560 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:14.559+0000 7f22a10ed540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:15.325 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:15.323+0000 7f22a10ed540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:16.112 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-09-30T00:46:16.145 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:16.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:16.231+0000 7f31c65fb540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-09-30T00:46:16.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:16.232+0000 7f31c65fb540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-09-30T00:46:16.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:16.232+0000 7f31c65fb540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T00:46:16.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:16.232+0000 7f31c65fb540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T00:46:16.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:16.232+0000 7f31c65fb540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T00:46:16.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:16.232+0000 7f31c65fb540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-09-30T00:46:16.764 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:16.763+0000 7f31c65fb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:17.528 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:17.527+0000 7f31c65fb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:18.293 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:18.293+0000 7f31c65fb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:19.081 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-09-30T00:46:19.113 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:19.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:19.201+0000 7f0013094540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-09-30T00:46:19.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:19.201+0000 7f0013094540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-09-30T00:46:19.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:19.201+0000 7f0013094540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T00:46:19.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:19.201+0000 7f0013094540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T00:46:19.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:19.201+0000 7f0013094540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T00:46:19.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:19.201+0000 7f0013094540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-09-30T00:46:19.725 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:19.724+0000 7f0013094540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:20.486 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:20.486+0000 7f0013094540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:21.249 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:21.249+0000 7f0013094540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:46:22.041 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-09-30T00:46:22.072 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-30T00:46:22.073 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:46:22.073 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-09-30T00:46:22.109 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:46:22.109 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-30T00:46:22.184 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:46:22.185 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-30T00:46:22.260 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:46:22.260 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-30T00:46:22.335 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:46:22.335 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-30T00:46:22.410 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:46:22.410 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-30T00:46:22.442 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:46:22.443 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-09-30T00:46:22.515 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:46:22.515 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-09-30T00:46:22.586 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:46:22.586 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-09-30T00:46:22.655 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:46:22.655 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-09-30T00:46:22.724 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:46:22.724 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-30T00:46:22.747 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:46:22.748 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-09-30T00:46:22.782 INFO:tasks.ceph:Adding keys to all mons... 2023-09-30T00:46:22.783 DEBUG:teuthology.orchestra.run.smithi040:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T00:46:22.790 DEBUG:teuthology.orchestra.run.smithi135:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T00:46:22.826 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.y] 2023-09-30T00:46:22.826 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQC/bxdlEBAqOhAAPx+5QCAzCw3LEdiiZH7sxg== 2023-09-30T00:46:22.826 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.0] 2023-09-30T00:46:22.827 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDDbxdlQJ/pFRAAGiTPKxLaAScr1mhdNt59+Q== 2023-09-30T00:46:22.827 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.1] 2023-09-30T00:46:22.827 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDGbxdlNV8GFBAAWv4rSY67s9qdPH6MMRYMqw== 2023-09-30T00:46:22.827 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.2] 2023-09-30T00:46:22.827 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDJbxdl/MfMEhAANKV7pcA1tMtfHrXvLvfCfQ== 2023-09-30T00:46:22.827 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.3] 2023-09-30T00:46:22.827 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDMbxdl4j2EERAA/dG/jmA3Y9qMoyDBJcdmtQ== 2023-09-30T00:46:22.827 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.x] 2023-09-30T00:46:22.827 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDAbxdlepxPAhAAr5q2ouQvthj2HH67DjlTPg== 2023-09-30T00:46:22.827 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.4] 2023-09-30T00:46:22.827 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDSbxdlaCwbEhAAnWvYlOA7BN1IJkDmKdzn1w== 2023-09-30T00:46:22.827 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.5] 2023-09-30T00:46:22.828 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDVbxdlAO35DxAAsAjhDE4GeAJJ3xICj1XEqw== 2023-09-30T00:46:22.828 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.6] 2023-09-30T00:46:22.828 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDYbxdlU3PpDRAAesJbp6OU0CghwHHUJY+JIg== 2023-09-30T00:46:22.828 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.7] 2023-09-30T00:46:22.828 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDbbxdl/Q4FDBAAAb6H0u8J61oY6NCGzZagiA== 2023-09-30T00:46:22.828 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2023-09-30T00:46:22.828 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDAbxdl+vWSBRAA52efU2/C8ryr6+RinQ7g7g== 2023-09-30T00:46:22.828 INFO:teuthology.orchestra.run.smithi040.stdout:[client.1] 2023-09-30T00:46:22.828 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDAbxdlpjPeCBAAEz4g9I+Y38IqpCRmoCNi9Q== 2023-09-30T00:46:22.849 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr.y] 2023-09-30T00:46:22.849 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQC/bxdlEBAqOhAAPx+5QCAzCw3LEdiiZH7sxg== 2023-09-30T00:46:22.849 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.0] 2023-09-30T00:46:22.850 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDDbxdlQJ/pFRAAGiTPKxLaAScr1mhdNt59+Q== 2023-09-30T00:46:22.850 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.1] 2023-09-30T00:46:22.850 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDGbxdlNV8GFBAAWv4rSY67s9qdPH6MMRYMqw== 2023-09-30T00:46:22.850 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.2] 2023-09-30T00:46:22.850 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDJbxdl/MfMEhAANKV7pcA1tMtfHrXvLvfCfQ== 2023-09-30T00:46:22.850 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.3] 2023-09-30T00:46:22.850 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDMbxdl4j2EERAA/dG/jmA3Y9qMoyDBJcdmtQ== 2023-09-30T00:46:22.850 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr.x] 2023-09-30T00:46:22.850 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDAbxdlepxPAhAAr5q2ouQvthj2HH67DjlTPg== 2023-09-30T00:46:22.850 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.4] 2023-09-30T00:46:22.850 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDSbxdlaCwbEhAAnWvYlOA7BN1IJkDmKdzn1w== 2023-09-30T00:46:22.850 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.5] 2023-09-30T00:46:22.851 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDVbxdlAO35DxAAsAjhDE4GeAJJ3xICj1XEqw== 2023-09-30T00:46:22.851 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.6] 2023-09-30T00:46:22.851 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDYbxdlU3PpDRAAesJbp6OU0CghwHHUJY+JIg== 2023-09-30T00:46:22.851 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.7] 2023-09-30T00:46:22.851 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDbbxdl/Q4FDBAAAb6H0u8J61oY6NCGzZagiA== 2023-09-30T00:46:22.851 INFO:teuthology.orchestra.run.smithi135.stdout:[client.0] 2023-09-30T00:46:22.851 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDAbxdl+vWSBRAA52efU2/C8ryr6+RinQ7g7g== 2023-09-30T00:46:22.851 INFO:teuthology.orchestra.run.smithi135.stdout:[client.1] 2023-09-30T00:46:22.851 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDAbxdlpjPeCBAAEz4g9I+Y38IqpCRmoCNi9Q== 2023-09-30T00:46:22.852 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:46:22.872 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:46:22.942 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:22.944 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.003 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.059 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.062 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.116 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.118 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.172 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:46:23.174 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:46:23.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.231 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.284 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.286 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.341 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.343 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.396 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.399 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:46:23.452 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:46:23.454 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:46:23.511 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:46:23.513 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:46:23.569 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-30T00:46:23.570 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-30T00:46:23.601 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:46:23.709 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-30T00:46:23.741 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-30T00:46:23.817 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:46:23.925 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-30T00:46:23.958 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-30T00:46:23.986 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:46:24.080 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-30T00:46:24.108 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:24.110 DEBUG:teuthology.orchestra.run.smithi135:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:46:24.166 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-30T00:46:24.166 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-09-30T00:46:24.166 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-30T00:46:24.167 DEBUG:teuthology.orchestra.run.smithi040:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-09-30T00:46:24.173 INFO:tasks.ceph.mon.a:Started 2023-09-30T00:46:24.173 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-09-30T00:46:24.173 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-30T00:46:24.173 DEBUG:teuthology.orchestra.run.smithi040:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-09-30T00:46:24.175 INFO:tasks.ceph.mon.c:Started 2023-09-30T00:46:24.175 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-09-30T00:46:24.176 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-30T00:46:24.176 DEBUG:teuthology.orchestra.run.smithi135:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-09-30T00:46:24.208 INFO:tasks.ceph.mon.b:Started 2023-09-30T00:46:24.208 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-30T00:46:24.209 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-09-30T00:46:24.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-09-30T00:46:24.211 INFO:tasks.ceph.mgr.y:Started 2023-09-30T00:46:24.211 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-30T00:46:24.211 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-09-30T00:46:24.213 INFO:tasks.ceph.mgr.x:Started 2023-09-30T00:46:24.213 INFO:tasks.ceph:Setting crush tunables to default 2023-09-30T00:46:24.214 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd crush tunables default 2023-09-30T00:46:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.350+0000 7f4434bb0700 10 monclient: get_monmap_and_config 2023-09-30T00:46:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.350+0000 7f4434bb0700 10 monclient: build_initial_monmap 2023-09-30T00:46:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.350+0000 7f4434bb0700 10 monclient: monmap: 2023-09-30T00:46:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:46:24.351 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:46:24.351726+0000 2023-09-30T00:46:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:46:24.351726+0000 2023-09-30T00:46:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:46:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:46:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:46:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:46:24.352 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:46:24.352 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:46:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.351+0000 7f4434bb0700 1 Processor -- start 2023-09-30T00:46:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.351+0000 7f4434bb0700 1 -- start start 2023-09-30T00:46:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.351+0000 7f4434bb0700 10 monclient: init 2023-09-30T00:46:24.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f4434bb0700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f4434bb0700 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-09-30T00:46:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f4434bb0700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f44300b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f4434bb0700 10 monclient(hunting): picked mon.noname-c con 0x7f44300b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:46:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f4434bb0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44300a5fb0 addr v1:172.21.15.40:6789/0 2023-09-30T00:46:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f4434bb0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f443013ac20 0x7f443013b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f4434bb0700 10 monclient(hunting): picked mon.noname-a con 0x7f443013ac20 addr v2:172.21.15.40:3300/0 2023-09-30T00:46:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f4434bb0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300a5b20 con 0x7f44300a5fb0 2023-09-30T00:46:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f4434bb0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f4434bb0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44300a71f0 con 0x7f443013ac20 2023-09-30T00:46:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f4434bb0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f4434bb0700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44300a74d0 con 0x7f44300b1460 2023-09-30T00:46:24.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f442dd9b700 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f44300a5fb0 legacy=0x7f4430141730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2023-09-30T00:46:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f4434bb0700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f4434bb0700 10 monclient(hunting): authenticate will time out at 5245.653404s 2023-09-30T00:46:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f442ed9d700 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f443013ac20 msgr2=0x7f443013b080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2023-09-30T00:46:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f442ed9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f443013ac20 0x7f443013b080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:46:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f442e59c700 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f44300b1830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2023-09-30T00:46:24.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.352+0000 7f442e59c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f44300b1830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:46:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.553+0000 7f442ed9d700 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f443013ac20 msgr2=0x7f443013b080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2023-09-30T00:46:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.553+0000 7f442dd9b700 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f44300a5fb0 legacy=0x7f4430141730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2023-09-30T00:46:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.553+0000 7f442ed9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f443013ac20 0x7f443013b080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:46:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.553+0000 7f442e59c700 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f44300b1830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2023-09-30T00:46:24.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.553+0000 7f442e59c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f44300b1830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:46:24.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.954+0000 7f442ed9d700 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f443013ac20 msgr2=0x7f443013b080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2023-09-30T00:46:24.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.954+0000 7f442dd9b700 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f44300a5fb0 legacy=0x7f4430141730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2023-09-30T00:46:24.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.954+0000 7f442ed9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f443013ac20 0x7f443013b080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:46:24.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.954+0000 7f442e59c700 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f44300b1830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2023-09-30T00:46:24.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:24.954+0000 7f442e59c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f44300b1830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:46:25.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:25.755+0000 7f442ed9d700 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f443013ac20 msgr2=0x7f443013b080 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2023-09-30T00:46:25.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:25.755+0000 7f442dd9b700 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f44300a5fb0 legacy=0x7f4430141730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2023-09-30T00:46:25.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:25.755+0000 7f442e59c700 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f44300b1830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2023-09-30T00:46:25.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:25.755+0000 7f442ed9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f443013ac20 0x7f443013b080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:46:25.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:25.755+0000 7f442e59c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f44300b1830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:46:27.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 10 monclient(hunting): tick 2023-09-30T00:46:27.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 1 monclient(hunting): continuing hunt 2023-09-30T00:46:27.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T00:46:27.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f44300b1830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:27.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f44300b1830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:27.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f443013ac20 msgr2=0x7f443013b080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:27.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f443013ac20 0x7f443013b080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:27.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f44300a5fb0 legacy=0x7f4430141730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:27.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 10 monclient: _add_conns ranks=[1,2] 2023-09-30T00:46:27.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4410003710 addr v1:172.21.15.135:6789/0 2023-09-30T00:46:27.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410007350 0x7f4410009800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:27.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f4410007350 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:27.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300a63c0 con 0x7f4410003710 2023-09-30T00:46:27.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:27.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44300a71f0 con 0x7f4410007350 2023-09-30T00:46:27.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f441ffff700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:27.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f442dd9b700 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410007350 msgr2=0x7f4410009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2023-09-30T00:46:27.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f442dd9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410007350 0x7f4410009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:46:27.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.352+0000 7f442e59c700 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f4410003710 legacy=0x7f4410005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.135:6789/0 2023-09-30T00:46:27.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.553+0000 7f442dd9b700 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410007350 msgr2=0x7f4410009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2023-09-30T00:46:27.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.553+0000 7f442dd9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410007350 0x7f4410009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:46:27.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.553+0000 7f442e59c700 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f4410003710 legacy=0x7f4410005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.135:6789/0 2023-09-30T00:46:27.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.954+0000 7f442dd9b700 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410007350 msgr2=0x7f4410009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2023-09-30T00:46:27.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.954+0000 7f442e59c700 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f4410003710 legacy=0x7f4410005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.135:6789/0 2023-09-30T00:46:27.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:27.954+0000 7f442dd9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410007350 0x7f4410009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:46:28.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:28.755+0000 7f442dd9b700 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410007350 msgr2=0x7f4410009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2023-09-30T00:46:28.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:28.755+0000 7f442e59c700 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f4410003710 legacy=0x7f4410005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.135:6789/0 2023-09-30T00:46:28.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:28.755+0000 7f442dd9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410007350 0x7f4410009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:46:30.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 10 monclient(hunting): tick 2023-09-30T00:46:30.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 1 monclient(hunting): continuing hunt 2023-09-30T00:46:30.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T00:46:30.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410007350 msgr2=0x7f4410009800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:30.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410007350 0x7f4410009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:30.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f4410003710 legacy=0x7f4410005b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:30.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-09-30T00:46:30.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f442ed9d700 1 -- reap_dead start 2023-09-30T00:46:30.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:30.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f44300b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:46:30.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441000a440 0x7f4410009800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:30.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f441000a440 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:30.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f441000a940 addr v1:172.21.15.135:6789/0 2023-09-30T00:46:30.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300a5b20 con 0x7f441000a940 2023-09-30T00:46:30.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:30.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44300a71f0 con 0x7f44300b1460 2023-09-30T00:46:30.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:30.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44300a74d0 con 0x7f441000a440 2023-09-30T00:46:30.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f441ffff700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:30.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f442dd9b700 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441000a440 msgr2=0x7f4410009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2023-09-30T00:46:30.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f442e59c700 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2023-09-30T00:46:30.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f442ed9d700 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f441000a940 legacy=0x7f4410005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.135:6789/0 2023-09-30T00:46:30.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f442dd9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441000a440 0x7f4410009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:46:30.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.352+0000 7f442e59c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:46:30.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.553+0000 7f442e59c700 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2023-09-30T00:46:30.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.553+0000 7f442ed9d700 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f441000a940 legacy=0x7f4410005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.135:6789/0 2023-09-30T00:46:30.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.553+0000 7f442dd9b700 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441000a440 msgr2=0x7f4410009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2023-09-30T00:46:30.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.553+0000 7f442e59c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:46:30.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.553+0000 7f442dd9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441000a440 0x7f4410009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:46:30.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.954+0000 7f442dd9b700 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441000a440 msgr2=0x7f4410009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2023-09-30T00:46:30.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.954+0000 7f442dd9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441000a440 0x7f4410009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:46:30.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.954+0000 7f442e59c700 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2023-09-30T00:46:30.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.954+0000 7f442ed9d700 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f441000a940 legacy=0x7f4410005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.135:6789/0 2023-09-30T00:46:30.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:30.954+0000 7f442e59c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:46:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:31.755+0000 7f442dd9b700 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441000a440 msgr2=0x7f4410009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3301/0 2023-09-30T00:46:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:31.755+0000 7f442dd9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441000a440 0x7f4410009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:46:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:31.755+0000 7f442e59c700 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.135:3300/0 2023-09-30T00:46:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:31.755+0000 7f442ed9d700 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f441000a940 legacy=0x7f4410005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.135:6789/0 2023-09-30T00:46:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:31.755+0000 7f442e59c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:46:33.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.352+0000 7f441ffff700 10 monclient(hunting): tick 2023-09-30T00:46:33.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.352+0000 7f441ffff700 1 monclient(hunting): continuing hunt 2023-09-30T00:46:33.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.352+0000 7f441ffff700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T00:46:33.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.352+0000 7f441ffff700 1 -- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441000a440 msgr2=0x7f4410009800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:33.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.352+0000 7f441ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f441000a440 0x7f4410009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:33.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.352+0000 7f441ffff700 1 -- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:33.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.352+0000 7f441ffff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:33.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.352+0000 7f441ffff700 1 -- >> v1:172.21.15.135:6789/0 conn(0x7f441000a940 legacy=0x7f4410005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:33.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.352+0000 7f441ffff700 10 monclient: _add_conns ranks=[0,3] 2023-09-30T00:46:33.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.353+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f441000d0a0 addr v1:172.21.15.40:6789/0 2023-09-30T00:46:33.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.353+0000 7f441ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f441000dc70 0x7f4410013230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:33.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.353+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f441000dc70 addr v2:172.21.15.40:3300/0 2023-09-30T00:46:33.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.353+0000 7f441ffff700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300a63c0 con 0x7f441000d0a0 2023-09-30T00:46:33.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.353+0000 7f441ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:33.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.353+0000 7f441ffff700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44300a71f0 con 0x7f441000dc70 2023-09-30T00:46:33.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.353+0000 7f441ffff700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:33.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.353+0000 7f442e59c700 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f441000d0a0 legacy=0x7f441000d4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2023-09-30T00:46:33.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.353+0000 7f442dd9b700 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f441000dc70 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2023-09-30T00:46:33.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.353+0000 7f442dd9b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f441000dc70 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:46:33.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.554+0000 7f442e59c700 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f441000d0a0 legacy=0x7f441000d4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2023-09-30T00:46:33.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.554+0000 7f442dd9b700 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f441000dc70 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2023-09-30T00:46:33.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.554+0000 7f442dd9b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f441000dc70 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:46:33.574 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:00:09.261 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:46:33.881 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:00:09.567 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:46:33.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.955+0000 7f442e59c700 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f441000d0a0 legacy=0x7f441000d4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2023-09-30T00:46:33.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.955+0000 7f442dd9b700 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f441000dc70 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2023-09-30T00:46:33.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:33.955+0000 7f442dd9b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f441000dc70 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:46:34.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:34.756+0000 7f442dd9b700 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f441000dc70 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2023-09-30T00:46:34.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:34.756+0000 7f442e59c700 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f441000d0a0 legacy=0x7f441000d4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2023-09-30T00:46:34.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:34.756+0000 7f442dd9b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f441000dc70 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:46:34.849 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:00:10.560 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:46:34.870 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:00:10.582 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:46:35.199 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:00:10.910 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:46:35.220 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:00:10.932 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:46:36.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 10 monclient(hunting): tick 2023-09-30T00:46:36.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 1 monclient(hunting): continuing hunt 2023-09-30T00:46:36.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T00:46:36.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f441000dc70 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:36.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f441000dc70 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:36.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f441000d0a0 legacy=0x7f441000d4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:36.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 10 monclient: _add_conns ranks=[4,3,1,0,2] 2023-09-30T00:46:36.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f442ed9d700 1 -- reap_dead start 2023-09-30T00:46:36.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:36.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f44300b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:46:36.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:36.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f441000a440 addr v2:172.21.15.40:3300/0 2023-09-30T00:46:36.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f441000d540 addr v1:172.21.15.135:6789/0 2023-09-30T00:46:36.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300a5b20 con 0x7f441000d540 2023-09-30T00:46:36.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:36.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44300a71f0 con 0x7f441000a440 2023-09-30T00:46:36.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:36.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44300a74d0 con 0x7f44300b1460 2023-09-30T00:46:36.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f441ffff700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:36.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f442dd9b700 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2023-09-30T00:46:36.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.353+0000 7f442dd9b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:46:36.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.436+0000 7f442e59c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:36.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.440+0000 7f442ed9d700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f441000d540 0x7f44100062b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:49468/0 (socket says 172.21.15.40:49468) 2023-09-30T00:46:36.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.440+0000 7f442ed9d700 1 -- 172.21.15.40:0/3183261509 learned_addr learned my addr 172.21.15.40:0/3183261509 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:46:36.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.441+0000 7f442e59c700 10 monclient(hunting): get_auth_request con 0x7f44300b1460 auth_method 0 2023-09-30T00:46:36.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.441+0000 7f442e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:36.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.441+0000 7f442e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:36.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.441+0000 7f442e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:36.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.444+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:36.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.444+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:36.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.444+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:36.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.444+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:46:36.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.645+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:36.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.645+0000 7f442e59c700 10 monclient(hunting): get_auth_request con 0x7f44300b1460 auth_method 0 2023-09-30T00:46:36.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.645+0000 7f442e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:36.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.645+0000 7f442e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:36.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.645+0000 7f442e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:36.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.646+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:46:36.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.646+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:36.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.646+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:36.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.646+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:46:36.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.861+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.898+0000 7f442dd9b700 10 monclient(hunting): get_auth_request con 0x7f441000a440 auth_method 0 2023-09-30T00:46:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.898+0000 7f442dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.898+0000 7f442dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:36.898+0000 7f442dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:37.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.004+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:37.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.004+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:37.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.004+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:37.006 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.004+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:46:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.057+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.057+0000 7f442e59c700 10 monclient(hunting): get_auth_request con 0x7f44300b1460 auth_method 0 2023-09-30T00:46:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.057+0000 7f442e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:37.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.057+0000 7f442e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.057+0000 7f442e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.058+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.058+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.058+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:37.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.058+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:46:37.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.463+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:37.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.463+0000 7f442dd9b700 10 monclient(hunting): get_auth_request con 0x7f441000a440 auth_method 0 2023-09-30T00:46:37.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.463+0000 7f442dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:37.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.463+0000 7f442dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:37.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.463+0000 7f442dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.465+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.465+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.465+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:37.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.465+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:46:37.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.859+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:37.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.859+0000 7f442e59c700 10 monclient(hunting): get_auth_request con 0x7f44300b1460 auth_method 0 2023-09-30T00:46:37.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.859+0000 7f442e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:37.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.859+0000 7f442e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:37.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.859+0000 7f442e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:37.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.860+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:37.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.860+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:37.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.860+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:37.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:37.860+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:46:38.303 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:38.303+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:38.308+0000 7f442dd9b700 10 monclient(hunting): get_auth_request con 0x7f441000a440 auth_method 0 2023-09-30T00:46:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:38.308+0000 7f442dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:38.308+0000 7f442dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:38.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:38.308+0000 7f442dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:38.316+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:38.316+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:38.316+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:38.316+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:46:39.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 10 monclient(hunting): tick 2023-09-30T00:46:39.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 1 monclient(hunting): continuing hunt 2023-09-30T00:46:39.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T00:46:39.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:39.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f44300b1460 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:39.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:39.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:39.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 >> v1:172.21.15.135:6789/0 conn(0x7f441000d540 legacy=0x7f44100062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:39.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 10 monclient: _add_conns ranks=[2,0] 2023-09-30T00:46:39.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:39.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f4410009800 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:39.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44100069c0 addr v1:172.21.15.40:6789/0 2023-09-30T00:46:39.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300a63c0 con 0x7f44100069c0 2023-09-30T00:46:39.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:39.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44300a71f0 con 0x7f4410009800 2023-09-30T00:46:39.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.353+0000 7f441ffff700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:39.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.354+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:39.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.354+0000 7f442e59c700 10 monclient(hunting): get_auth_request con 0x7f4410009800 auth_method 0 2023-09-30T00:46:39.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.354+0000 7f442e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:39.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.354+0000 7f442e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:39.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.354+0000 7f442e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:39.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.355+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:39.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.355+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:39.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.355+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:39.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.355+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:46:39.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.556+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:39.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.557+0000 7f442e59c700 10 monclient(hunting): get_auth_request con 0x7f4410009800 auth_method 0 2023-09-30T00:46:39.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.557+0000 7f442e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:39.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.557+0000 7f442e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:39.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.557+0000 7f442e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:39.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.558+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:39.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.558+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:39.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.558+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:39.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.558+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:46:39.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.959+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:39.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.959+0000 7f442e59c700 10 monclient(hunting): get_auth_request con 0x7f4410009800 auth_method 0 2023-09-30T00:46:39.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.959+0000 7f442e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:39.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.959+0000 7f442e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:39.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.959+0000 7f442e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:39.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.960+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:39.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.960+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:39.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.960+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:39.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:39.960+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:46:40.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:40.762+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:40.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:40.762+0000 7f442e59c700 10 monclient(hunting): get_auth_request con 0x7f4410009800 auth_method 0 2023-09-30T00:46:40.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:40.762+0000 7f442e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:40.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:40.762+0000 7f442e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:40.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:40.762+0000 7f442e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:40.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:40.762+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:40.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:40.762+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:40.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:40.762+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:40.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:40.762+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:46:42.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 10 monclient(hunting): tick 2023-09-30T00:46:42.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 1 monclient(hunting): continuing hunt 2023-09-30T00:46:42.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T00:46:42.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:42.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:42.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 >> v1:172.21.15.40:6789/0 conn(0x7f44100069c0 legacy=0x7f44100101f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:42.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-09-30T00:46:42.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f442ed9d700 1 -- 172.21.15.40:0/3183261509 reap_dead start 2023-09-30T00:46:42.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:42.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f441000a440 addr v2:172.21.15.40:3300/0 2023-09-30T00:46:42.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:42.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f4410009800 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:42.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4410010900 addr v1:172.21.15.40:6789/0 2023-09-30T00:46:42.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4410006dc0 con 0x7f4410010900 2023-09-30T00:46:42.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:42.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44300a71f0 con 0x7f441000a440 2023-09-30T00:46:42.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:42.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44300a74d0 con 0x7f4410009800 2023-09-30T00:46:42.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f441ffff700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:42.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.354+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:42.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.355+0000 7f442dd9b700 10 monclient(hunting): get_auth_request con 0x7f4410009800 auth_method 0 2023-09-30T00:46:42.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.355+0000 7f442dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:42.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.355+0000 7f442dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:42.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.355+0000 7f442dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:42.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.355+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:46:42.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.355+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:42.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.355+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:42.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.355+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:42.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.355+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:46:42.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.357+0000 7f442e59c700 10 monclient(hunting): get_auth_request con 0x7f441000a440 auth_method 0 2023-09-30T00:46:42.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.357+0000 7f442e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:42.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.357+0000 7f442e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:42.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.357+0000 7f442e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:42.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.358+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:42.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.358+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:42.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.358+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:42.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.358+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:46:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.557+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:42.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.557+0000 7f442dd9b700 10 monclient(hunting): get_auth_request con 0x7f4410009800 auth_method 0 2023-09-30T00:46:42.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.557+0000 7f442dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:42.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.557+0000 7f442dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:42.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.557+0000 7f442dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:42.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.559+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:42.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.559+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:42.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.559+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:42.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.559+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:46:42.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.559+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:42.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.559+0000 7f442e59c700 10 monclient(hunting): get_auth_request con 0x7f441000a440 auth_method 0 2023-09-30T00:46:42.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.559+0000 7f442e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:42.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.559+0000 7f442e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:42.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.559+0000 7f442e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.560+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.560+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.560+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.560+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:46:42.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.960+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:42.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.961+0000 7f442dd9b700 10 monclient(hunting): get_auth_request con 0x7f4410009800 auth_method 0 2023-09-30T00:46:42.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.961+0000 7f442dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:42.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.961+0000 7f442dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:42.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.961+0000 7f442dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:42.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.961+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:42.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.961+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:42.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.962+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:42.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.962+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:42.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.962+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:46:42.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.962+0000 7f442e59c700 10 monclient(hunting): get_auth_request con 0x7f441000a440 auth_method 0 2023-09-30T00:46:42.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.962+0000 7f442e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:42.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.962+0000 7f442e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:42.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.962+0000 7f442e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:42.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.963+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:46:42.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.963+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:42.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.963+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:42.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:42.963+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:46:43.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.764+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:43.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.764+0000 7f442dd9b700 10 monclient(hunting): get_auth_request con 0x7f4410009800 auth_method 0 2023-09-30T00:46:43.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.764+0000 7f442dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:43.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.764+0000 7f442dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:43.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.764+0000 7f442dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:43.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.764+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:43.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.764+0000 7f442e59c700 10 monclient(hunting): get_auth_request con 0x7f441000a440 auth_method 0 2023-09-30T00:46:43.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.764+0000 7f442e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.764+0000 7f442e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.764+0000 7f442e59c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.766+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:46:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.766+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.766+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:43.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.766+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:46:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.768+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:46:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.768+0000 7f442e59c700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:43.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.768+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:43.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:43.768+0000 7f442e59c700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:46:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 10 monclient(hunting): tick 2023-09-30T00:46:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 1 monclient(hunting): continuing hunt 2023-09-30T00:46:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T00:46:45.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 msgr2=0x7f4410005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:45.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4410009800 0x7f4410005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:45.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 msgr2=0x7f4410013230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:45.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.40:3300/0 conn(0x7f441000a440 0x7f4410013230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:45.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 >> v1:172.21.15.40:6789/0 conn(0x7f4410010900 legacy=0x7f44100179c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:45.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 10 monclient: _add_conns ranks=[1,4] 2023-09-30T00:46:45.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f441000d540 addr v1:172.21.15.135:6789/0 2023-09-30T00:46:45.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 0x7f4410009f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:45.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f4410010d10 addr v2:172.21.15.135:3300/0 2023-09-30T00:46:45.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4410007040 con 0x7f441000d540 2023-09-30T00:46:45.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:45.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 1 -- 172.21.15.40:0/3183261509 --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44300a71f0 con 0x7f4410010d10 2023-09-30T00:46:45.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.355+0000 7f441ffff700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:45.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.356+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 0x7f4410009f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:45.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.356+0000 7f442dd9b700 10 monclient(hunting): get_auth_request con 0x7f4410010d10 auth_method 0 2023-09-30T00:46:45.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.356+0000 7f442dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:45.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.356+0000 7f442dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:45.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.356+0000 7f442dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:45.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.357+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:46:45.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.357+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:45.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.357+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 0x7f4410009f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:45.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.358+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:46:45.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.559+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 0x7f4410009f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.560+0000 7f442dd9b700 10 monclient(hunting): get_auth_request con 0x7f4410010d10 auth_method 0 2023-09-30T00:46:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.560+0000 7f442dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.560+0000 7f442dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.560+0000 7f442dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.561+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:46:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.561+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:45.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.561+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 0x7f4410009f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:45.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.561+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:46:45.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.962+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 0x7f4410009f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:45.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.963+0000 7f442dd9b700 10 monclient(hunting): get_auth_request con 0x7f4410010d10 auth_method 0 2023-09-30T00:46:45.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.963+0000 7f442dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:45.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.963+0000 7f442dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:45.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.963+0000 7f442dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:45.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.964+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:46:45.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.964+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:45.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.964+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 0x7f4410009f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:45.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:45.964+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:46:46.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:46.765+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 0x7f4410009f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:46.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:46.766+0000 7f442dd9b700 10 monclient(hunting): get_auth_request con 0x7f4410010d10 auth_method 0 2023-09-30T00:46:46.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:46.766+0000 7f442dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:46.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:46.766+0000 7f442dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:46.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:46.766+0000 7f442dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:46.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:46.769+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:46:46.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:46.769+0000 7f442dd9b700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:46:46.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:46.769+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 0x7f4410009f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:46.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:46.769+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:46:47.475 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:47.473+0000 7feafc14a1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T00:46:47.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.564+0000 7f442d59a700 1 -- 172.21.15.40:0/3183261509 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538751279 0 0) 0x7f4410007040 con 0x7f441000d540 2023-09-30T00:46:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.564+0000 7f442d59a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.564+0000 7f442d59a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.564+0000 7f442d59a700 10 monclient(hunting): my global_id is 4105 2023-09-30T00:46:47.566 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.564+0000 7f442d59a700 1 -- 172.21.15.40:0/3183261509 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4418003570 con 0x7f441000d540 2023-09-30T00:46:47.641 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:47.640+0000 7feafc14a1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T00:46:47.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 1 -- 172.21.15.40:0/3183261509 <== mon.1 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1700190191 0 0) 0x7f4418003570 con 0x7f441000d540 2023-09-30T00:46:47.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 1 -- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 msgr2=0x7f4410009f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:46:47.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 1 --2- 172.21.15.40:0/3183261509 >> v2:172.21.15.135:3300/0 conn(0x7f4410010d10 0x7f4410009f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:47.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient: _finish_hunting 0 2023-09-30T00:46:47.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 1 monclient: found mon.noname-c-legacy 2023-09-30T00:46:47.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:47.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.135:6789/0 2023-09-30T00:46:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 1 -- 172.21.15.40:0/3183261509 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441000e1f0 con 0x7f441000d540 2023-09-30T00:46:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient: _finish_auth 0 2023-09-30T00:46:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 1 -- 172.21.15.40:0/3183261509 <== mon.1 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3033000540 0 0) 0x7f4420007a60 con 0x7f441000d540 2023-09-30T00:46:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-09-30T00:46:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient: dump: 2023-09-30T00:46:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:46:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:47.662 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:47.663 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:47.663 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:47.663 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:47.663 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:47.663 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:46:47.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] but i'm connected to v1:172.21.15.135:6789/0, reconnecting 2023-09-30T00:46:47.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:47.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 1 -- 172.21.15.40:0/3183261509 >> v1:172.21.15.135:6789/0 conn(0x7f441000d540 legacy=0x7f44100062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:47.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:46:47.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442ed9d700 1 -- 172.21.15.40:0/3183261509 reap_dead start 2023-09-30T00:46:47.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4418003a60 0x7f4418003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:47.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient(hunting): picked mon.b con 0x7f4418003a60 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:46:47.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4418004810 0x7f4418004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient(hunting): picked mon.a con 0x7f4418004810 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:46:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4418005210 0x7f441800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient(hunting): picked mon.c con 0x7f4418005210 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 1 -- 172.21.15.40:0/3183261509 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44300a71f0 con 0x7f4418004810 2023-09-30T00:46:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 1 -- 172.21.15.40:0/3183261509 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441800c1b0 con 0x7f4418003a60 2023-09-30T00:46:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 1 -- 172.21.15.40:0/3183261509 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441800c370 con 0x7f4418005210 2023-09-30T00:46:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:47.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.659+0000 7f442d59a700 10 monclient(hunting): _finish_auth 0 2023-09-30T00:46:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.661+0000 7f442ed9d700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4418005210 0x7f441800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.662+0000 7f442ed9d700 10 monclient(hunting): get_auth_request con 0x7f4418005210 auth_method 0 2023-09-30T00:46:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.662+0000 7f442ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:47.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.662+0000 7f442ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.662+0000 7f442ed9d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.663+0000 7f442ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.663+0000 7f442ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.663+0000 7f442ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:46:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.664+0000 7f442dd9b700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4418004810 0x7f4418004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.665+0000 7f442dd9b700 10 monclient(hunting): get_auth_request con 0x7f4418004810 auth_method 0 2023-09-30T00:46:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.665+0000 7f442dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.665+0000 7f442dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:47.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.665+0000 7f442dd9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.666+0000 7f442dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.666+0000 7f442dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.666+0000 7f442dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:46:47.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.671+0000 7f442ed9d700 10 monclient(hunting): handle_auth_done global_id 4105 payload 293 2023-09-30T00:46:47.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.671+0000 7f442ed9d700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4418003a60 msgr2=0x7f4418003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:47.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.671+0000 7f442ed9d700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4418003a60 0x7f4418003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:47.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.671+0000 7f442ed9d700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4418004810 msgr2=0x7f4418004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:47.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.671+0000 7f442ed9d700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4418004810 0x7f4418004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:47.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.671+0000 7f442ed9d700 10 monclient: _finish_hunting 0 2023-09-30T00:46:47.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.671+0000 7f442ed9d700 1 monclient: found mon.c 2023-09-30T00:46:47.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.671+0000 7f442ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:47.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.671+0000 7f442ed9d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:46:47.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.671+0000 7f442ed9d700 1 -- 172.21.15.40:0/3183261509 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4418003570 con 0x7f4418005210 2023-09-30T00:46:47.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.671+0000 7f442ed9d700 10 monclient: _finish_auth 0 2023-09-30T00:46:47.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.671+0000 7f442ed9d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:47.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.671+0000 7f4434bb0700 5 monclient: authenticate success, global_id 4105 2023-09-30T00:46:47.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.671+0000 7f4434bb0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:47.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.677+0000 7f442ed9d700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4418005210 0x7f441800ba60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f443006f560 tx=0x7f4428009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6af736d36c85c99a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.759+0000 7f442d59a700 1 -- 172.21.15.40:0/3183261509 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428009640 con 0x7f4418005210 2023-09-30T00:46:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.759+0000 7f442d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.759+0000 7f442d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.759+0000 7f442d59a700 10 monclient: dump: 2023-09-30T00:46:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:46:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:47.761 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:46:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.759+0000 7f442d59a700 1 -- 172.21.15.40:0/3183261509 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4428003bd0 con 0x7f4418005210 2023-09-30T00:46:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.759+0000 7f4434bb0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.759+0000 7f442d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.759+0000 7f4434bb0700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.759+0000 7f442d59a700 1 -- 172.21.15.40:0/3183261509 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428009ee0 con 0x7f4418005210 2023-09-30T00:46:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.759+0000 7f4434bb0700 4 set_mon_vals no callback set 2023-09-30T00:46:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.759+0000 7f442d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.759+0000 7f442d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.759+0000 7f442d59a700 10 monclient: dump: 2023-09-30T00:46:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:46:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:47.763 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:47.763 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:46:47.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.760+0000 7f4434bb0700 10 monclient: shutdown 2023-09-30T00:46:47.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.760+0000 7f4434bb0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:47.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.760+0000 7f4434bb0700 1 -- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4418005210 msgr2=0x7f441800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:47.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.760+0000 7f4434bb0700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4418005210 0x7f441800ba60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f443006f560 tx=0x7f4428009310 comp rx=0 tx=0).stop 2023-09-30T00:46:47.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.760+0000 7f4434bb0700 1 -- 172.21.15.40:0/3183261509 shutdown_connections 2023-09-30T00:46:47.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.760+0000 7f4434bb0700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4418005210 0x7f441800ba60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:47.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.760+0000 7f4434bb0700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4418004810 0x7f4418004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:47.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.760+0000 7f4434bb0700 1 --2- 172.21.15.40:0/3183261509 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4418003a60 0x7f4418003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:47.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.760+0000 7f4434bb0700 1 -- 172.21.15.40:0/3183261509 >> 172.21.15.40:0/3183261509 conn(0x7f44300631f0 msgr2=0x7f44300635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:47.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.760+0000 7f4434bb0700 1 -- 172.21.15.40:0/3183261509 shutdown_connections 2023-09-30T00:46:47.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.760+0000 7f4434bb0700 1 -- 172.21.15.40:0/3183261509 wait complete. 2023-09-30T00:46:47.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.761+0000 7f4434bb0700 10 monclient: build_initial_monmap 2023-09-30T00:46:47.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.761+0000 7f4434bb0700 10 monclient: monmap: 2023-09-30T00:46:47.767 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:46:47.767 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:47.767 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:46:47.767 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:46:47.767 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:47.767 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:46:47.767 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:46:47.767 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:46:47.767 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:46:47.767 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:46:47.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.761+0000 7f4434bb0700 1 librados: starting msgr at 2023-09-30T00:46:47.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.761+0000 7f4434bb0700 1 librados: starting objecter 2023-09-30T00:46:47.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.761+0000 7f4434bb0700 1 Processor -- start 2023-09-30T00:46:47.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.761+0000 7f4434bb0700 1 -- start start 2023-09-30T00:46:47.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.761+0000 7f4434bb0700 1 librados: setting wanted keys 2023-09-30T00:46:47.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.761+0000 7f4434bb0700 1 librados: calling monclient init 2023-09-30T00:46:47.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.761+0000 7f4434bb0700 10 monclient: init 2023-09-30T00:46:47.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:47.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:46:47.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4418003a60 0x7f44300ac470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:47.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 10 monclient(hunting): picked mon.noname-b con 0x7f4418003a60 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:46:47.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4418004810 0x7f4430139af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:47.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 10 monclient(hunting): picked mon.noname-c con 0x7f4418004810 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:47.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4418005210 0x7f443013a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:47.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 10 monclient(hunting): picked mon.noname-a con 0x7f4418005210 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:46:47.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:47.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44300ae910 con 0x7f4418005210 2023-09-30T00:46:47.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:47.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44300ad610 con 0x7f4418003a60 2023-09-30T00:46:47.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:47.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44300ab670 con 0x7f4418004810 2023-09-30T00:46:47.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:47.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.762+0000 7f4434bb0700 10 monclient(hunting): authenticate will time out at 5269.063582s 2023-09-30T00:46:47.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.763+0000 7f442ed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4418005210 0x7f443013a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:47.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.763+0000 7f442e59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4418003a60 0x7f44300ac470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:47.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.763+0000 7f442ed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4418005210 0x7f443013a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:43614/0 (socket says 172.21.15.40:43614) 2023-09-30T00:46:47.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.763+0000 7f442ed9d700 1 -- 172.21.15.40:0/3247255844 learned_addr learned my addr 172.21.15.40:0/3247255844 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:46:47.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.763+0000 7f442ed9d700 10 monclient(hunting): get_auth_request con 0x7f4418005210 auth_method 0 2023-09-30T00:46:47.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.763+0000 7f442ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:47.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.763+0000 7f442ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:47.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.763+0000 7f442ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:47.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.763+0000 7f442e59c700 10 monclient(hunting): get_auth_request con 0x7f4418003a60 auth_method 0 2023-09-30T00:46:47.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.763+0000 7f442e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:47.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.763+0000 7f442e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:47.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.763+0000 7f442e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:47.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.764+0000 7f442ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:47.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.764+0000 7f442ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:47.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.764+0000 7f442ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:47.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.764+0000 7f442e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:47.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.764+0000 7f442e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:47.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.764+0000 7f442e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:47.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.767+0000 7f442e59c700 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2023-09-30T00:46:47.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.767+0000 7f442e59c700 1 -- 172.21.15.40:0/3247255844 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4418004810 msgr2=0x7f4430139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:47.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.767+0000 7f442e59c700 1 --2- 172.21.15.40:0/3247255844 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4418004810 0x7f4430139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:47.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.767+0000 7f442e59c700 1 -- 172.21.15.40:0/3247255844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4418005210 msgr2=0x7f443013a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:47.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.767+0000 7f442e59c700 1 --2- 172.21.15.40:0/3247255844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4418005210 0x7f443013a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:47.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.767+0000 7f442e59c700 10 monclient: _finish_hunting 0 2023-09-30T00:46:47.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.767+0000 7f442e59c700 1 monclient: found mon.noname-b 2023-09-30T00:46:47.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.767+0000 7f442e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:47.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.767+0000 7f442e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:46:47.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.767+0000 7f442e59c700 1 -- 172.21.15.40:0/3247255844 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443013a570 con 0x7f4418003a60 2023-09-30T00:46:47.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.769+0000 7f442e59c700 1 --2- 172.21.15.40:0/3247255844 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4418003a60 0x7f44300ac470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4420003ee0 tx=0x7f4420007af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:47.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.769+0000 7f441e7fc700 10 client.?.objecter ms_handle_connect 0x7f4418003a60 2023-09-30T00:46:47.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.769+0000 7f441e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:46:47.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f441e7fc700 1 -- 172.21.15.40:0/3247255844 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4420013bc0 con 0x7f4418003a60 2023-09-30T00:46:47.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f441e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:47.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f441e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:47.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f441e7fc700 10 monclient: dump: 2023-09-30T00:46:47.775 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:46:47.775 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:47.775 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:47.775 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:47.776 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:47.776 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:47.776 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:47.776 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:47.776 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:47.776 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:46:47.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f441e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:46:47.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f441e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:47.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f441e7fc700 1 -- 172.21.15.40:0/3247255844 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442001a040 con 0x7f4418003a60 2023-09-30T00:46:47.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f4434bb0700 5 monclient: authenticate success, global_id 4111 2023-09-30T00:46:47.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f4434bb0700 10 monclient: _renew_subs 2023-09-30T00:46:47.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f4434bb0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:47.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f4434bb0700 1 -- 172.21.15.40:0/3247255844 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44300aed30 con 0x7f4418003a60 2023-09-30T00:46:47.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f441e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:47.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f441e7fc700 1 -- 172.21.15.40:0/3247255844 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442001f440 con 0x7f4418003a60 2023-09-30T00:46:47.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f441e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:47.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f441e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:47.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f441e7fc700 10 monclient: dump: 2023-09-30T00:46:47.777 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:46:47.777 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:47.777 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:47.778 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:47.778 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:47.778 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:47.778 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:47.778 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:47.778 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:47.778 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:46:47.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f442cd99700 4 set_mon_vals no callback set 2023-09-30T00:46:47.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f4434bb0700 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:46:47.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f4434bb0700 10 monclient: _renew_subs 2023-09-30T00:46:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f4434bb0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f4434bb0700 1 -- 172.21.15.40:0/3247255844 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44300ad300 con 0x7f4418003a60 2023-09-30T00:46:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.774+0000 7f4434bb0700 1 librados: init done 2023-09-30T00:46:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.775+0000 7f4434bb0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.775+0000 7f4434bb0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.775+0000 7f4434bb0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.775+0000 7f4434bb0700 1 -- 172.21.15.40:0/3247255844 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443014d220 con 0x7f4418003a60 2023-09-30T00:46:47.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.841+0000 7f441e7fc700 1 -- 172.21.15.40:0/3247255844 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f442001f830 con 0x7f4418003a60 2023-09-30T00:46:47.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.841+0000 7f441e7fc700 1 -- 172.21.15.40:0/3247255844 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f442000b040 con 0x7f4418003a60 2023-09-30T00:46:47.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.841+0000 7f441e7fc700 10 client.4111.objecter ms_dispatch 0x7f4430001550 osd_map(1..1 src has 1..1) v4 2023-09-30T00:46:47.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.841+0000 7f441e7fc700 3 client.4111.objecter handle_osd_map got epochs [1,1] > 0 2023-09-30T00:46:47.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.841+0000 7f441e7fc700 3 client.4111.objecter handle_osd_map decoding full epoch 1 2023-09-30T00:46:47.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.841+0000 7f441e7fc700 20 client.4111.objecter dump_active .. 0 homeless 2023-09-30T00:46:47.849 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:47.848+0000 7fadf12501c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T00:46:47.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.972+0000 7f441e7fc700 1 -- 172.21.15.40:0/3247255844 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4420009040 con 0x7f4418003a60 2023-09-30T00:46:47.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.972+0000 7f441e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:47.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:47.972+0000 7f441e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:47.981 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:47.980+0000 7feafc14a1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T00:46:48.074 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:48.073+0000 7fadf12501c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T00:46:48.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.129+0000 7f4434bb0700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T00:46:48.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.129+0000 7f4434bb0700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T00:46:48.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.129+0000 7f4434bb0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:48.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.129+0000 7f4434bb0700 1 -- 172.21.15.40:0/3247255844 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f44300aea50 con 0x7f4418003a60 2023-09-30T00:46:48.470 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:48.469+0000 7fadf12501c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T00:46:48.508 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:48.507+0000 7feafc14a1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T00:46:48.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.578+0000 7f441e7fc700 1 -- 172.21.15.40:0/3247255844 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f44300aea50 con 0x7f4418003a60 2023-09-30T00:46:48.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.578+0000 7f441e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T00:46:48.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.578+0000 7f441e7fc700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-09-30T00:46:48.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.578+0000 7f4434bb0700 10 librados: watch_flush enter 2023-09-30T00:46:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.578+0000 7f4434bb0700 10 librados: watch_flush exit 2023-09-30T00:46:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.578+0000 7f4434bb0700 20 client.4111.objecter shutdown clearing up homeless session... 2023-09-30T00:46:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.578+0000 7f4434bb0700 10 client.4111.objecter successfully canceled tick 2023-09-30T00:46:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.579+0000 7f4434bb0700 10 monclient: shutdown 2023-09-30T00:46:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.579+0000 7f4434bb0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.579+0000 7f4434bb0700 1 -- 172.21.15.40:0/3247255844 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4418003a60 msgr2=0x7f44300ac470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.579+0000 7f4434bb0700 1 --2- 172.21.15.40:0/3247255844 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4418003a60 0x7f44300ac470 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4420003ee0 tx=0x7f4420007af0 comp rx=0 tx=0).stop 2023-09-30T00:46:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.580+0000 7f4434bb0700 1 -- 172.21.15.40:0/3247255844 shutdown_connections 2023-09-30T00:46:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.580+0000 7f4434bb0700 1 --2- 172.21.15.40:0/3247255844 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4418003a60 0x7f44300ac470 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.580+0000 7f4434bb0700 1 --2- 172.21.15.40:0/3247255844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4418005210 0x7f443013a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:48.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.580+0000 7f4434bb0700 1 --2- 172.21.15.40:0/3247255844 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4418004810 0x7f4430139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:48.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.580+0000 7f4434bb0700 1 -- 172.21.15.40:0/3247255844 >> 172.21.15.40:0/3247255844 conn(0x7f44300631f0 msgr2=0x7f443013b060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:48.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.580+0000 7f4434bb0700 1 -- 172.21.15.40:0/3247255844 shutdown_connections 2023-09-30T00:46:48.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.581+0000 7f4434bb0700 1 -- 172.21.15.40:0/3247255844 wait complete. 2023-09-30T00:46:48.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:48.581+0000 7f4434bb0700 1 librados: shutdown 2023-09-30T00:46:48.583 INFO:teuthology.orchestra.run.smithi040.stderr:adjusted tunables profile to default 2023-09-30T00:46:48.596 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-30T00:46:48.596 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:46:48.596 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-30T00:46:48.628 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:46:48.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-30T00:46:48.648 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:48.647+0000 7feafc14a1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T00:46:48.699 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:46:48.699 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-30T00:46:48.770 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2023-09-30T00:46:48.770 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-30T00:46:48.841 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:46:48.841 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-09-30T00:46:48.869 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:46:48.869 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-09-30T00:46:48.936 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:46:48.936 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-09-30T00:46:48.989 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:48.988+0000 7feafc14a1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T00:46:49.002 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-09-30T00:46:49.002 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-09-30T00:46:49.057 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:49.057+0000 7fadf12501c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T00:46:49.070 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 66e1da65-310e-4463-8aa0-431896832e64 0 2023-09-30T00:46:49.224 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:49.224+0000 7fadf12501c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T00:46:49.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.233+0000 7f949a556700 10 monclient: get_monmap_and_config 2023-09-30T00:46:49.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.233+0000 7f949a556700 10 monclient: build_initial_monmap 2023-09-30T00:46:49.235 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.233+0000 7f949a556700 10 monclient: monmap: 2023-09-30T00:46:49.235 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:49.235 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:49.235 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:46:49.235116+0000 2023-09-30T00:46:49.235 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:46:49.235116+0000 2023-09-30T00:46:49.235 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:49.235 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:49.235 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:46:49.235 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:46:49.235 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:46:49.235 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:46:49.236 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:46:49.236 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:49.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 1 Processor -- start 2023-09-30T00:46:49.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 1 -- start start 2023-09-30T00:46:49.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 10 monclient: init 2023-09-30T00:46:49.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:49.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-09-30T00:46:49.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94940b1460 0x7f94940b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 10 monclient(hunting): picked mon.noname-b con 0x7f94940b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:49.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f94940a5fb0 0x7f9494141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 10 monclient(hunting): picked mon.noname-c con 0x7f94940a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:46:49.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94940af410 addr v1:172.21.15.40:6789/0 2023-09-30T00:46:49.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94940a71f0 con 0x7f94940af410 2023-09-30T00:46:49.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:49.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94940a74d0 con 0x7f94940a5fb0 2023-09-30T00:46:49.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:49.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94940a63c0 con 0x7f94940b1460 2023-09-30T00:46:49.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:49.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.234+0000 7f949a556700 10 monclient(hunting): authenticate will time out at 5277.673961s 2023-09-30T00:46:49.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.235+0000 7f94937fe700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f94940a5fb0 0x7f9494141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.235+0000 7f9498af3700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f94940af410 0x7f94940af890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.135:51886/0 (socket says 172.21.15.135:51886) 2023-09-30T00:46:49.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.235+0000 7f9498af3700 1 -- 172.21.15.135:0/1843847148 learned_addr learned my addr 172.21.15.135:0/1843847148 (peer_addr_for_me v1:172.21.15.135:0/0) 2023-09-30T00:46:49.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.235+0000 7f9493fff700 1 --2- 172.21.15.135:0/1843847148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94940b1460 0x7f94940b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.236+0000 7f94937fe700 10 monclient(hunting): get_auth_request con 0x7f94940a5fb0 auth_method 0 2023-09-30T00:46:49.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.236+0000 7f94937fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:49.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.236+0000 7f94937fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:49.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.236+0000 7f94937fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:49.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.236+0000 7f9493fff700 10 monclient(hunting): get_auth_request con 0x7f94940b1460 auth_method 0 2023-09-30T00:46:49.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.236+0000 7f9493fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:49.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.236+0000 7f9493fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:49.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.236+0000 7f9493fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:49.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.236+0000 7f94937fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:49.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.236+0000 7f94937fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:49.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.236+0000 7f94937fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:49.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.236+0000 7f9493fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:49.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.236+0000 7f9493fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:49.241 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.236+0000 7f9493fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:49.241 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.238+0000 7f94937fe700 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2023-09-30T00:46:49.241 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.238+0000 7f94937fe700 1 -- 172.21.15.135:0/1843847148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94940b1460 msgr2=0x7f94940b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:49.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.238+0000 7f94937fe700 1 --2- 172.21.15.135:0/1843847148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94940b1460 0x7f94940b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.238+0000 7f94937fe700 1 -- 172.21.15.135:0/1843847148 >> v1:172.21.15.40:6789/0 conn(0x7f94940af410 legacy=0x7f94940af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:49.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.238+0000 7f94937fe700 10 monclient: _finish_hunting 0 2023-09-30T00:46:49.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.238+0000 7f94937fe700 1 monclient: found mon.noname-c 2023-09-30T00:46:49.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.238+0000 7f94937fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:49.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.238+0000 7f94937fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:46:49.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.238+0000 7f94937fe700 1 -- 172.21.15.135:0/1843847148 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94940a56d0 con 0x7f94940a5fb0 2023-09-30T00:46:49.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.240+0000 7f94937fe700 1 --2- 172.21.15.135:0/1843847148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94940a5fb0 0x7f9494141730 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f94880098e0 tx=0x7f94880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12cfd95d4c50027a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:49.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.242+0000 7f9492ffd700 1 -- 172.21.15.135:0/1843847148 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948800b640 con 0x7f94940a5fb0 2023-09-30T00:46:49.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.242+0000 7f9492ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:49.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.242+0000 7f9492ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:49.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.242+0000 7f9492ffd700 10 monclient: dump: 2023-09-30T00:46:49.244 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:49.244 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:49.244 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.244 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.244 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:49.244 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:49.244 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:49.244 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:49.245 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:49.245 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:49.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.242+0000 7f9492ffd700 10 monclient: _finish_auth 0 2023-09-30T00:46:49.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.242+0000 7f9492ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:49.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.242+0000 7f949a556700 5 monclient: authenticate success, global_id 4114 2023-09-30T00:46:49.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.242+0000 7f949a556700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:49.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.244+0000 7f9492ffd700 1 -- 172.21.15.135:0/1843847148 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948800b7a0 con 0x7f94940a5fb0 2023-09-30T00:46:49.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.244+0000 7f9492ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:49.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.244+0000 7f9492ffd700 1 -- 172.21.15.135:0/1843847148 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948800bb40 con 0x7f94940a5fb0 2023-09-30T00:46:49.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.244+0000 7f949a556700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:49.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.244+0000 7f949a556700 4 set_mon_vals no callback set 2023-09-30T00:46:49.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.244+0000 7f9492ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:49.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.244+0000 7f9492ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:49.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.244+0000 7f9492ffd700 10 monclient: dump: 2023-09-30T00:46:49.247 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:49.247 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:49.247 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.247 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.247 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:49.247 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:49.247 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:49.248 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:49.248 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:49.248 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:49.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.245+0000 7f949a556700 10 monclient: shutdown 2023-09-30T00:46:49.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.245+0000 7f949a556700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:49.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.245+0000 7f949a556700 1 -- 172.21.15.135:0/1843847148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94940a5fb0 msgr2=0x7f9494141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:49.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.245+0000 7f949a556700 1 --2- 172.21.15.135:0/1843847148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94940a5fb0 0x7f9494141730 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f94880098e0 tx=0x7f94880091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.245+0000 7f949a556700 1 -- 172.21.15.135:0/1843847148 shutdown_connections 2023-09-30T00:46:49.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.245+0000 7f949a556700 1 --2- 172.21.15.135:0/1843847148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94940b1460 0x7f94940b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.245+0000 7f949a556700 1 --2- 172.21.15.135:0/1843847148 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94940a5fb0 0x7f9494141730 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.245+0000 7f949a556700 1 -- 172.21.15.135:0/1843847148 >> 172.21.15.135:0/1843847148 conn(0x7f94940631f0 msgr2=0x7f94940635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:49.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.245+0000 7f949a556700 1 -- 172.21.15.135:0/1843847148 shutdown_connections 2023-09-30T00:46:49.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.245+0000 7f949a556700 1 -- 172.21.15.135:0/1843847148 wait complete. 2023-09-30T00:46:49.251 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.245+0000 7f949a556700 10 monclient: build_initial_monmap 2023-09-30T00:46:49.251 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.245+0000 7f949a556700 10 monclient: monmap: 2023-09-30T00:46:49.251 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:49.251 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:49.252 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2023-09-30T00:46:49.252 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2023-09-30T00:46:49.252 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:49.252 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2023-09-30T00:46:49.252 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:46:49.252 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:46:49.252 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:46:49.252 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:49.252 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.246+0000 7f949a556700 1 librados: starting msgr at 2023-09-30T00:46:49.252 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.246+0000 7f949a556700 1 librados: starting objecter 2023-09-30T00:46:49.252 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.246+0000 7f949a556700 1 Processor -- start 2023-09-30T00:46:49.252 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.246+0000 7f949a556700 1 -- start start 2023-09-30T00:46:49.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.246+0000 7f949a556700 1 librados: setting wanted keys 2023-09-30T00:46:49.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.246+0000 7f949a556700 1 librados: calling monclient init 2023-09-30T00:46:49.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.246+0000 7f949a556700 10 monclient: init 2023-09-30T00:46:49.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:49.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:46:49.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94940a5fb0 0x7f9494151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 10 monclient(hunting): picked mon.noname-b con 0x7f94940a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:46:49.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94940af410 0x7f94940a7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 10 monclient(hunting): picked mon.noname-c con 0x7f94940af410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:49.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94940b1460 0x7f94940a8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 10 monclient(hunting): picked mon.noname-a con 0x7f94940b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:46:49.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:49.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94940a9d70 con 0x7f94940b1460 2023-09-30T00:46:49.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:49.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94940a8050 con 0x7f94940a5fb0 2023-09-30T00:46:49.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:49.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f949413c830 con 0x7f94940af410 2023-09-30T00:46:49.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:49.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f949a556700 10 monclient(hunting): authenticate will time out at 5277.686244s 2023-09-30T00:46:49.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f9493fff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94940a5fb0 0x7f9494151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.247+0000 7f9498af3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94940b1460 0x7f94940a8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9493fff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94940a5fb0 0x7f9494151a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:54130/0 (socket says 172.21.15.135:54130) 2023-09-30T00:46:49.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9493fff700 1 -- 172.21.15.135:0/1786110413 learned_addr learned my addr 172.21.15.135:0/1786110413 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-09-30T00:46:49.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9493fff700 10 monclient(hunting): get_auth_request con 0x7f94940a5fb0 auth_method 0 2023-09-30T00:46:49.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9493fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:49.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9493fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:49.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9493fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:49.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f94937fe700 1 --2- 172.21.15.135:0/1786110413 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94940af410 0x7f94940a7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9498af3700 10 monclient(hunting): get_auth_request con 0x7f94940b1460 auth_method 0 2023-09-30T00:46:49.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9498af3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:49.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9498af3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:49.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9498af3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:49.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f94937fe700 10 monclient(hunting): get_auth_request con 0x7f94940af410 auth_method 0 2023-09-30T00:46:49.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f94937fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:49.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f94937fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:49.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f94937fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:49.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9493fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:49.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9493fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:49.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9493fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:49.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9498af3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:49.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9498af3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:49.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.248+0000 7f9498af3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:49.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.249+0000 7f94937fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:49.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.249+0000 7f94937fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:49.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.249+0000 7f94937fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:49.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.251+0000 7f9493fff700 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2023-09-30T00:46:49.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.251+0000 7f9493fff700 1 -- 172.21.15.135:0/1786110413 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94940af410 msgr2=0x7f94940a7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:49.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.251+0000 7f9493fff700 1 --2- 172.21.15.135:0/1786110413 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94940af410 0x7f94940a7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.251+0000 7f9493fff700 1 -- 172.21.15.135:0/1786110413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94940b1460 msgr2=0x7f94940a8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:49.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.251+0000 7f9493fff700 1 --2- 172.21.15.135:0/1786110413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94940b1460 0x7f94940a8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.251+0000 7f9493fff700 10 monclient: _finish_hunting 0 2023-09-30T00:46:49.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.251+0000 7f9493fff700 1 monclient: found mon.noname-b 2023-09-30T00:46:49.258 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.251+0000 7f9493fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:49.258 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.251+0000 7f9493fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:46:49.258 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.251+0000 7f9493fff700 1 -- 172.21.15.135:0/1786110413 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9488009420 con 0x7f94940a5fb0 2023-09-30T00:46:49.258 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.253+0000 7f9493fff700 1 --2- 172.21.15.135:0/1786110413 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94940a5fb0 0x7f9494151a00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f948400e930 tx=0x7f94840091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:49.258 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.253+0000 7f947bfff700 10 client.?.objecter ms_handle_connect 0x7f94940a5fb0 2023-09-30T00:46:49.258 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.253+0000 7f947bfff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:46:49.259 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f947bfff700 1 -- 172.21.15.135:0/1786110413 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9484013680 con 0x7f94940a5fb0 2023-09-30T00:46:49.259 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f947bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:49.259 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f947bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:49.259 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f947bfff700 10 monclient: dump: 2023-09-30T00:46:49.259 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:49.259 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:49.259 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.260 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.260 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:49.260 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:49.260 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:49.260 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:49.260 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:49.260 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:49.260 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f947bfff700 10 monclient: _finish_auth 0 2023-09-30T00:46:49.260 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f947bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:49.260 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f947bfff700 1 -- 172.21.15.135:0/1786110413 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94840137e0 con 0x7f94940a5fb0 2023-09-30T00:46:49.260 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f949a556700 5 monclient: authenticate success, global_id 4117 2023-09-30T00:46:49.260 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f949a556700 10 monclient: _renew_subs 2023-09-30T00:46:49.261 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f949a556700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:49.261 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f949a556700 1 -- 172.21.15.135:0/1786110413 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9494152450 con 0x7f94940a5fb0 2023-09-30T00:46:49.261 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f947bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:49.261 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f947bfff700 1 -- 172.21.15.135:0/1786110413 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9484013b80 con 0x7f94940a5fb0 2023-09-30T00:46:49.261 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f947bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:49.261 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f947bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:49.261 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f947bfff700 10 monclient: dump: 2023-09-30T00:46:49.261 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:49.261 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:49.261 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.261 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.261 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:49.261 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:49.262 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:49.262 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:49.262 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:49.262 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:49.262 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f94927fc700 4 set_mon_vals no callback set 2023-09-30T00:46:49.262 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f949a556700 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:46:49.262 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f949a556700 10 monclient: _renew_subs 2023-09-30T00:46:49.262 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f949a556700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:49.262 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f949a556700 1 -- 172.21.15.135:0/1786110413 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94940a8c70 con 0x7f94940a5fb0 2023-09-30T00:46:49.262 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.257+0000 7f949a556700 1 librados: init done 2023-09-30T00:46:49.263 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.258+0000 7f949a556700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:49.263 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.258+0000 7f949a556700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:49.263 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.258+0000 7f949a556700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:49.263 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.258+0000 7f949a556700 1 -- 172.21.15.135:0/1786110413 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94940a7ec0 con 0x7f94940a5fb0 2023-09-30T00:46:49.263 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.260+0000 7f947bfff700 1 -- 172.21.15.135:0/1786110413 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9484005c80 con 0x7f94940a5fb0 2023-09-30T00:46:49.285 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.284+0000 7f947bfff700 1 -- 172.21.15.135:0/1786110413 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f94840068c0 con 0x7f94940a5fb0 2023-09-30T00:46:49.285 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.284+0000 7f947bfff700 10 client.4117.objecter ms_dispatch 0x7f9494001550 osd_map(2..2 src has 1..2) v4 2023-09-30T00:46:49.285 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.284+0000 7f947bfff700 3 client.4117.objecter handle_osd_map got epochs [2,2] > 0 2023-09-30T00:46:49.285 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.284+0000 7f947bfff700 3 client.4117.objecter handle_osd_map decoding full epoch 2 2023-09-30T00:46:49.285 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.284+0000 7f947bfff700 20 client.4117.objecter dump_active .. 0 homeless 2023-09-30T00:46:49.368 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.367+0000 7f947bfff700 1 -- 172.21.15.135:0/1786110413 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9484021060 con 0x7f94940a5fb0 2023-09-30T00:46:49.368 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.367+0000 7f947bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:49.368 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.367+0000 7f947bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:49.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.448+0000 7f949a556700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "66e1da65-310e-4463-8aa0-431896832e64", "id": 0}] 2023-09-30T00:46:49.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.448+0000 7f949a556700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "66e1da65-310e-4463-8aa0-431896832e64", "id": 0}] 2023-09-30T00:46:49.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.448+0000 7f949a556700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:49.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.448+0000 7f949a556700 1 -- 172.21.15.135:0/1786110413 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "66e1da65-310e-4463-8aa0-431896832e64", "id": 0} v 0) v1 -- 0x7f94940a8ec0 con 0x7f94940a5fb0 2023-09-30T00:46:49.609 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:49.609+0000 7fadf12501c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T00:46:49.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.637+0000 7f947bfff700 1 -- 172.21.15.135:0/1786110413 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "66e1da65-310e-4463-8aa0-431896832e64", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9484017070 con 0x7f94940a5fb0 2023-09-30T00:46:49.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.637+0000 7f947bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "66e1da65-310e-4463-8aa0-431896832e64", "id": 0}] 2023-09-30T00:46:49.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.637+0000 7f947bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:49.639 INFO:teuthology.orchestra.run.smithi135.stdout:0 2023-09-30T00:46:49.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 10 librados: watch_flush enter 2023-09-30T00:46:49.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 10 librados: watch_flush exit 2023-09-30T00:46:49.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 20 client.4117.objecter shutdown clearing up homeless session... 2023-09-30T00:46:49.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 10 client.4117.objecter successfully canceled tick 2023-09-30T00:46:49.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 10 monclient: shutdown 2023-09-30T00:46:49.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:49.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 1 -- 172.21.15.135:0/1786110413 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94940a5fb0 msgr2=0x7f9494151a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:49.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 1 --2- 172.21.15.135:0/1786110413 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94940a5fb0 0x7f9494151a00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f948400e930 tx=0x7f94840091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 1 -- 172.21.15.135:0/1786110413 shutdown_connections 2023-09-30T00:46:49.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 1 --2- 172.21.15.135:0/1786110413 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f94940a5fb0 0x7f9494151a00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 1 --2- 172.21.15.135:0/1786110413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94940b1460 0x7f94940a8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 1 --2- 172.21.15.135:0/1786110413 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f94940af410 0x7f94940a7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 1 -- 172.21.15.135:0/1786110413 >> 172.21.15.135:0/1786110413 conn(0x7f94940631f0 msgr2=0x7f94941399c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:49.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 1 -- 172.21.15.135:0/1786110413 shutdown_connections 2023-09-30T00:46:49.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.638+0000 7f949a556700 1 -- 172.21.15.135:0/1786110413 wait complete. 2023-09-30T00:46:49.642 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.639+0000 7f949a556700 1 librados: shutdown 2023-09-30T00:46:49.653 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new d6dd269f-2eda-4ef7-9da1-2ed5b16291b4 1 2023-09-30T00:46:49.741 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:49.740+0000 7feafc14a1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T00:46:49.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.768+0000 7f98712e2700 10 monclient: get_monmap_and_config 2023-09-30T00:46:49.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.768+0000 7f98712e2700 10 monclient: build_initial_monmap 2023-09-30T00:46:49.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.768+0000 7f98712e2700 10 monclient: monmap: 2023-09-30T00:46:49.770 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:49.770 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:49.771 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:46:49.770336+0000 2023-09-30T00:46:49.771 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:46:49.770336+0000 2023-09-30T00:46:49.771 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:49.771 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:49.771 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:46:49.771 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:46:49.771 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:46:49.771 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:46:49.771 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:46:49.771 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:49.771 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 1 Processor -- start 2023-09-30T00:46:49.771 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 1 -- start start 2023-09-30T00:46:49.771 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 10 monclient: init 2023-09-30T00:46:49.773 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:49.773 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-09-30T00:46:49.774 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f986c0b1460 0x7f986c0b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.774 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 10 monclient(hunting): picked mon.noname-c con 0x7f986c0b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:46:49.774 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f986c0a5fb0 0x7f986c141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.774 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 10 monclient(hunting): picked mon.noname-b con 0x7f986c0a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:49.774 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f986c0af410 addr v1:172.21.15.40:6789/0 2023-09-30T00:46:49.774 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986c0a71f0 con 0x7f986c0af410 2023-09-30T00:46:49.774 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:49.774 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f986c0a74d0 con 0x7f986c0b1460 2023-09-30T00:46:49.774 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:49.774 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f986c0a63c0 con 0x7f986c0a5fb0 2023-09-30T00:46:49.774 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:49.774 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.769+0000 7f98712e2700 10 monclient(hunting): authenticate will time out at 5278.208986s 2023-09-30T00:46:49.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.770+0000 7f986affd700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f986c0b1460 0x7f986c0b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.770+0000 7f986b7fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f986c0af410 0x7f986c0af890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.135:51888/0 (socket says 172.21.15.135:51888) 2023-09-30T00:46:49.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.770+0000 7f986b7fe700 1 -- 172.21.15.135:0/3363815481 learned_addr learned my addr 172.21.15.135:0/3363815481 (peer_addr_for_me v1:172.21.15.135:0/0) 2023-09-30T00:46:49.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.770+0000 7f986a7fc700 1 --2- 172.21.15.135:0/3363815481 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f986c0a5fb0 0x7f986c141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.771+0000 7f986affd700 10 monclient(hunting): get_auth_request con 0x7f986c0b1460 auth_method 0 2023-09-30T00:46:49.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.771+0000 7f986affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:49.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.771+0000 7f986affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:49.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.771+0000 7f986affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:49.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.771+0000 7f986a7fc700 10 monclient(hunting): get_auth_request con 0x7f986c0a5fb0 auth_method 0 2023-09-30T00:46:49.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.771+0000 7f986a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:49.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.771+0000 7f986a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:49.776 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.771+0000 7f986a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:49.776 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.771+0000 7f986affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:49.776 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.771+0000 7f986affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:49.776 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.771+0000 7f986affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:49.776 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.771+0000 7f986a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:49.776 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.771+0000 7f986a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:49.776 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.771+0000 7f986a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:49.777 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.773+0000 7f986affd700 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2023-09-30T00:46:49.777 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.773+0000 7f986affd700 1 -- 172.21.15.135:0/3363815481 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f986c0a5fb0 msgr2=0x7f986c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:49.777 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.773+0000 7f986affd700 1 --2- 172.21.15.135:0/3363815481 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f986c0a5fb0 0x7f986c141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.777 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.773+0000 7f986affd700 1 -- 172.21.15.135:0/3363815481 >> v1:172.21.15.40:6789/0 conn(0x7f986c0af410 legacy=0x7f986c0af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:49.777 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.773+0000 7f986affd700 10 monclient: _finish_hunting 0 2023-09-30T00:46:49.777 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.773+0000 7f986affd700 1 monclient: found mon.noname-c 2023-09-30T00:46:49.777 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.773+0000 7f986affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:49.777 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.773+0000 7f986affd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:46:49.777 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.773+0000 7f986affd700 1 -- 172.21.15.135:0/3363815481 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c0a56d0 con 0x7f986c0b1460 2023-09-30T00:46:49.777 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.775+0000 7f986affd700 1 --2- 172.21.15.135:0/3363815481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f986c0b1460 0x7f986c0b1830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f985c0098e0 tx=0x7f985c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe201040dd99cc98 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:49.780 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f9869ffb700 1 -- 172.21.15.135:0/3363815481 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c00b640 con 0x7f986c0b1460 2023-09-30T00:46:49.781 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f9869ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:49.781 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f9869ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:49.781 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f9869ffb700 10 monclient: dump: 2023-09-30T00:46:49.781 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:49.781 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:49.781 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.781 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.782 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:49.782 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:49.782 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:49.782 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:49.782 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:49.782 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:49.782 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f9869ffb700 10 monclient: _finish_auth 0 2023-09-30T00:46:49.782 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f9869ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:49.782 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f9869ffb700 1 -- 172.21.15.135:0/3363815481 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985c00b7a0 con 0x7f986c0b1460 2023-09-30T00:46:49.782 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f9869ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:49.783 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f9869ffb700 1 -- 172.21.15.135:0/3363815481 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c00bb40 con 0x7f986c0b1460 2023-09-30T00:46:49.783 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f98712e2700 5 monclient: authenticate success, global_id 4120 2023-09-30T00:46:49.783 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f98712e2700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:49.783 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f98712e2700 4 set_mon_vals no callback set 2023-09-30T00:46:49.783 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f9869ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:49.783 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f9869ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:49.783 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.779+0000 7f9869ffb700 10 monclient: dump: 2023-09-30T00:46:49.783 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:49.784 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:49.784 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.784 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.784 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:49.784 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:49.784 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:49.784 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:49.784 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:49.784 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:49.788 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.780+0000 7f98712e2700 10 monclient: shutdown 2023-09-30T00:46:49.788 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.780+0000 7f98712e2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:49.788 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.780+0000 7f98712e2700 1 -- 172.21.15.135:0/3363815481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f986c0b1460 msgr2=0x7f986c0b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:49.788 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.780+0000 7f98712e2700 1 --2- 172.21.15.135:0/3363815481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f986c0b1460 0x7f986c0b1830 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f985c0098e0 tx=0x7f985c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.788 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.780+0000 7f98712e2700 1 -- 172.21.15.135:0/3363815481 shutdown_connections 2023-09-30T00:46:49.788 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.780+0000 7f98712e2700 1 --2- 172.21.15.135:0/3363815481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f986c0b1460 0x7f986c0b1830 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.788 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.780+0000 7f98712e2700 1 --2- 172.21.15.135:0/3363815481 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f986c0a5fb0 0x7f986c141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.789 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.780+0000 7f98712e2700 1 -- 172.21.15.135:0/3363815481 >> 172.21.15.135:0/3363815481 conn(0x7f986c0631f0 msgr2=0x7f986c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:49.789 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.780+0000 7f98712e2700 1 -- 172.21.15.135:0/3363815481 shutdown_connections 2023-09-30T00:46:49.789 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.780+0000 7f98712e2700 1 -- 172.21.15.135:0/3363815481 wait complete. 2023-09-30T00:46:49.789 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.781+0000 7f98712e2700 10 monclient: build_initial_monmap 2023-09-30T00:46:49.789 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.781+0000 7f98712e2700 10 monclient: monmap: 2023-09-30T00:46:49.789 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:49.789 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:49.789 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2023-09-30T00:46:49.789 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2023-09-30T00:46:49.789 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:49.789 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2023-09-30T00:46:49.790 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:46:49.790 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:46:49.790 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:46:49.790 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:49.790 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.781+0000 7f98712e2700 1 librados: starting msgr at 2023-09-30T00:46:49.790 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.781+0000 7f98712e2700 1 librados: starting objecter 2023-09-30T00:46:49.790 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.782+0000 7f98712e2700 1 Processor -- start 2023-09-30T00:46:49.790 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.782+0000 7f98712e2700 1 -- start start 2023-09-30T00:46:49.790 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.782+0000 7f98712e2700 1 librados: setting wanted keys 2023-09-30T00:46:49.790 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.782+0000 7f98712e2700 1 librados: calling monclient init 2023-09-30T00:46:49.791 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.782+0000 7f98712e2700 10 monclient: init 2023-09-30T00:46:49.791 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.782+0000 7f98712e2700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:49.791 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.782+0000 7f98712e2700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:46:49.791 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.782+0000 7f98712e2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f986c0a5fb0 0x7f986c13dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.791 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f98712e2700 10 monclient(hunting): picked mon.noname-b con 0x7f986c0a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:46:49.791 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f98712e2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f986c0af410 0x7f986c0a7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.791 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f98712e2700 10 monclient(hunting): picked mon.noname-c con 0x7f986c0af410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:49.791 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f98712e2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f986c0b1460 0x7f986c0a8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.791 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f98712e2700 10 monclient(hunting): picked mon.noname-a con 0x7f986c0b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:46:49.791 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f98712e2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:49.791 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f98712e2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986c0a8950 con 0x7f986c0b1460 2023-09-30T00:46:49.792 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f98712e2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:49.792 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f98712e2700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986c0a7a20 con 0x7f986c0a5fb0 2023-09-30T00:46:49.792 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f98712e2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:49.792 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f98712e2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f986c0a8fd0 con 0x7f986c0af410 2023-09-30T00:46:49.792 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f98712e2700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:49.792 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f98712e2700 10 monclient(hunting): authenticate will time out at 5278.222078s 2023-09-30T00:46:49.792 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f986affd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f986c0a5fb0 0x7f986c13dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.792 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f986b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f986c0b1460 0x7f986c0a8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.792 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f986a7fc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f986c0af410 0x7f986c0a7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.792 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f986affd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f986c0a5fb0 0x7f986c13dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:54152/0 (socket says 172.21.15.135:54152) 2023-09-30T00:46:49.792 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f986affd700 1 -- 172.21.15.135:0/1700836552 learned_addr learned my addr 172.21.15.135:0/1700836552 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-09-30T00:46:49.793 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f986affd700 10 monclient(hunting): get_auth_request con 0x7f986c0a5fb0 auth_method 0 2023-09-30T00:46:49.793 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f986affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:49.793 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f986affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:49.793 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.783+0000 7f986affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:49.793 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986b7fe700 10 monclient(hunting): get_auth_request con 0x7f986c0b1460 auth_method 0 2023-09-30T00:46:49.793 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:49.793 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:49.793 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:49.793 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986a7fc700 10 monclient(hunting): get_auth_request con 0x7f986c0af410 auth_method 0 2023-09-30T00:46:49.793 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:49.793 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:49.794 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:49.794 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:49.794 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:49.794 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:49.794 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:49.794 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:49.794 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:49.794 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:49.794 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:49.794 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.784+0000 7f986a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:49.796 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.787+0000 7f986affd700 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2023-09-30T00:46:49.796 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.787+0000 7f986affd700 1 -- 172.21.15.135:0/1700836552 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f986c0af410 msgr2=0x7f986c0a7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:49.796 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.787+0000 7f986affd700 1 --2- 172.21.15.135:0/1700836552 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f986c0af410 0x7f986c0a7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.796 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.787+0000 7f986affd700 1 -- 172.21.15.135:0/1700836552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f986c0b1460 msgr2=0x7f986c0a8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:49.796 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.787+0000 7f986affd700 1 --2- 172.21.15.135:0/1700836552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f986c0b1460 0x7f986c0a8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.787+0000 7f986affd700 10 monclient: _finish_hunting 0 2023-09-30T00:46:49.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.787+0000 7f986affd700 1 monclient: found mon.noname-b 2023-09-30T00:46:49.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.787+0000 7f986affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:49.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.787+0000 7f986affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:46:49.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.787+0000 7f986affd700 1 -- 172.21.15.135:0/1700836552 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c009420 con 0x7f986c0a5fb0 2023-09-30T00:46:49.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.789+0000 7f986affd700 1 --2- 172.21.15.135:0/1700836552 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f986c0a5fb0 0x7f986c13dfe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f986c06f5e0 tx=0x7f985c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:49.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.789+0000 7f9852ffd700 10 client.?.objecter ms_handle_connect 0x7f986c0a5fb0 2023-09-30T00:46:49.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.789+0000 7f9852ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:46:49.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f9852ffd700 1 -- 172.21.15.135:0/1700836552 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c01bb70 con 0x7f986c0a5fb0 2023-09-30T00:46:49.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f9852ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:49.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f9852ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:49.798 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f9852ffd700 10 monclient: dump: 2023-09-30T00:46:49.798 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:49.798 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:49.798 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.798 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.798 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:49.798 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:49.798 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:49.798 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:49.798 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:49.798 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:49.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f9852ffd700 10 monclient: _finish_auth 0 2023-09-30T00:46:49.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f9852ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:49.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f9852ffd700 1 -- 172.21.15.135:0/1700836552 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f985c01bcd0 con 0x7f986c0a5fb0 2023-09-30T00:46:49.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f98712e2700 5 monclient: authenticate success, global_id 4123 2023-09-30T00:46:49.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f98712e2700 10 monclient: _renew_subs 2023-09-30T00:46:49.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f98712e2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:49.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f98712e2700 1 -- 172.21.15.135:0/1700836552 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f986c14de40 con 0x7f986c0a5fb0 2023-09-30T00:46:49.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f9852ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:49.801 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f9852ffd700 1 -- 172.21.15.135:0/1700836552 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985c011670 con 0x7f986c0a5fb0 2023-09-30T00:46:49.801 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f9852ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:49.801 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f9852ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:49.801 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f9852ffd700 10 monclient: dump: 2023-09-30T00:46:49.801 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:49.801 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:49.801 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.801 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:49.802 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:49.802 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:49.802 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:49.802 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:49.802 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:49.802 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:49.802 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f98697fa700 4 set_mon_vals no callback set 2023-09-30T00:46:49.802 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f98712e2700 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:46:49.802 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f98712e2700 10 monclient: _renew_subs 2023-09-30T00:46:49.802 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f98712e2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:49.802 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f98712e2700 1 -- 172.21.15.135:0/1700836552 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f986c14e350 con 0x7f986c0a5fb0 2023-09-30T00:46:49.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.793+0000 7f98712e2700 1 librados: init done 2023-09-30T00:46:49.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.794+0000 7f98712e2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:49.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.794+0000 7f98712e2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:49.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.794+0000 7f98712e2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:49.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.794+0000 7f98712e2700 1 -- 172.21.15.135:0/1700836552 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f986c064a00 con 0x7f986c0a5fb0 2023-09-30T00:46:49.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.796+0000 7f9852ffd700 1 -- 172.21.15.135:0/1700836552 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f985c0119e0 con 0x7f986c0a5fb0 2023-09-30T00:46:49.818 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:49.816+0000 7feafc14a1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T00:46:49.903 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.902+0000 7f9852ffd700 1 -- 172.21.15.135:0/1700836552 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f985c02c460 con 0x7f986c0a5fb0 2023-09-30T00:46:49.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.902+0000 7f9852ffd700 10 client.4123.objecter ms_dispatch 0x7f986c001550 osd_map(3..3 src has 1..3) v4 2023-09-30T00:46:49.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.902+0000 7f9852ffd700 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2023-09-30T00:46:49.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.902+0000 7f9852ffd700 3 client.4123.objecter handle_osd_map decoding full epoch 3 2023-09-30T00:46:49.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.902+0000 7f9852ffd700 20 client.4123.objecter dump_active .. 0 homeless 2023-09-30T00:46:49.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.902+0000 7f9852ffd700 1 -- 172.21.15.135:0/1700836552 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f985c015020 con 0x7f986c0a5fb0 2023-09-30T00:46:49.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.902+0000 7f9852ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:49.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.902+0000 7f9852ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:49.956 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:49.955+0000 7feafc14a1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T00:46:49.986 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.984+0000 7f98712e2700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d6dd269f-2eda-4ef7-9da1-2ed5b16291b4", "id": 1}] 2023-09-30T00:46:49.986 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.984+0000 7f98712e2700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d6dd269f-2eda-4ef7-9da1-2ed5b16291b4", "id": 1}] 2023-09-30T00:46:49.986 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.984+0000 7f98712e2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:49.986 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:49.984+0000 7f98712e2700 1 -- 172.21.15.135:0/1700836552 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d6dd269f-2eda-4ef7-9da1-2ed5b16291b4", "id": 1} v 0) v1 -- 0x7f986c0b42c0 con 0x7f986c0a5fb0 2023-09-30T00:46:50.026 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:50.024+0000 7feafc14a1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T00:46:50.085 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.084+0000 7f9852ffd700 1 -- 172.21.15.135:0/1700836552 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d6dd269f-2eda-4ef7-9da1-2ed5b16291b4", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f985c030020 con 0x7f986c0a5fb0 2023-09-30T00:46:50.085 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.084+0000 7f9852ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d6dd269f-2eda-4ef7-9da1-2ed5b16291b4", "id": 1}] 2023-09-30T00:46:50.085 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.084+0000 7f9852ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:50.086 INFO:teuthology.orchestra.run.smithi135.stdout:1 2023-09-30T00:46:50.087 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 10 librados: watch_flush enter 2023-09-30T00:46:50.087 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 10 librados: watch_flush exit 2023-09-30T00:46:50.087 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 20 client.4123.objecter shutdown clearing up homeless session... 2023-09-30T00:46:50.087 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 10 client.4123.objecter successfully canceled tick 2023-09-30T00:46:50.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 10 monclient: shutdown 2023-09-30T00:46:50.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:50.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 1 -- 172.21.15.135:0/1700836552 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f986c0a5fb0 msgr2=0x7f986c13dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:50.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 1 --2- 172.21.15.135:0/1700836552 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f986c0a5fb0 0x7f986c13dfe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f986c06f5e0 tx=0x7f985c017000 comp rx=0 tx=0).stop 2023-09-30T00:46:50.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 1 -- 172.21.15.135:0/1700836552 shutdown_connections 2023-09-30T00:46:50.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 1 --2- 172.21.15.135:0/1700836552 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f986c0a5fb0 0x7f986c13dfe0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 1 --2- 172.21.15.135:0/1700836552 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f986c0b1460 0x7f986c0a8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 1 --2- 172.21.15.135:0/1700836552 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f986c0af410 0x7f986c0a7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 1 -- 172.21.15.135:0/1700836552 >> 172.21.15.135:0/1700836552 conn(0x7f986c0631f0 msgr2=0x7f986c142fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:50.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 1 -- 172.21.15.135:0/1700836552 shutdown_connections 2023-09-30T00:46:50.089 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 1 -- 172.21.15.135:0/1700836552 wait complete. 2023-09-30T00:46:50.089 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.085+0000 7f98712e2700 1 librados: shutdown 2023-09-30T00:46:50.099 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 8f0de27d-3674-4ff4-8451-4fcc90786c98 2 2023-09-30T00:46:50.218 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:50.217+0000 7feafc14a1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T00:46:50.221 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.220+0000 7ffa1b6b6700 10 monclient: get_monmap_and_config 2023-09-30T00:46:50.221 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.220+0000 7ffa1b6b6700 10 monclient: build_initial_monmap 2023-09-30T00:46:50.222 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.220+0000 7ffa1b6b6700 10 monclient: monmap: 2023-09-30T00:46:50.222 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:50.222 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:50.222 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:46:50.222255+0000 2023-09-30T00:46:50.222 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:46:50.222255+0000 2023-09-30T00:46:50.222 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:50.222 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:50.222 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:46:50.222 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:46:50.222 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:46:50.223 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:46:50.223 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:46:50.223 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:50.225 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.221+0000 7ffa1b6b6700 1 Processor -- start 2023-09-30T00:46:50.225 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.221+0000 7ffa1b6b6700 1 -- start start 2023-09-30T00:46:50.225 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.221+0000 7ffa1b6b6700 10 monclient: init 2023-09-30T00:46:50.225 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.221+0000 7ffa1b6b6700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:50.225 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.221+0000 7ffa1b6b6700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-09-30T00:46:50.225 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.221+0000 7ffa1b6b6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffa140a5710 0x7ffa140a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:50.225 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.221+0000 7ffa1b6b6700 10 monclient(hunting): picked mon.noname-a con 0x7ffa140a5710 addr v2:172.21.15.40:3300/0 2023-09-30T00:46:50.225 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.221+0000 7ffa1b6b6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa140a7b20 addr v1:172.21.15.40:6789/0 2023-09-30T00:46:50.225 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.221+0000 7ffa1b6b6700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ffa1413e8e0 0x7ffa1413ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:50.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.222+0000 7ffa1b6b6700 10 monclient(hunting): picked mon.noname-c con 0x7ffa1413e8e0 addr v2:172.21.15.135:3300/0 2023-09-30T00:46:50.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.222+0000 7ffa1b6b6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa140a7ff0 con 0x7ffa140a7b20 2023-09-30T00:46:50.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.222+0000 7ffa1b6b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:50.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.222+0000 7ffa1b6b6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa140aedc0 con 0x7ffa140a5710 2023-09-30T00:46:50.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.222+0000 7ffa1b6b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:50.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.222+0000 7ffa1b6b6700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa140aef00 con 0x7ffa1413e8e0 2023-09-30T00:46:50.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.222+0000 7ffa1b6b6700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:50.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.222+0000 7ffa1b6b6700 10 monclient(hunting): authenticate will time out at 5278.661051s 2023-09-30T00:46:50.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.222+0000 7ffa19c53700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ffa1413e8e0 0x7ffa1413ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:50.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.223+0000 7ffa19452700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffa140a5710 0x7ffa140a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:50.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.223+0000 7ffa18c51700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ffa140a7b20 0x7ffa1413e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.135:51898/0 (socket says 172.21.15.135:51898) 2023-09-30T00:46:50.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.223+0000 7ffa18c51700 1 -- 172.21.15.135:0/2976029894 learned_addr learned my addr 172.21.15.135:0/2976029894 (peer_addr_for_me v1:172.21.15.135:0/0) 2023-09-30T00:46:50.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.223+0000 7ffa19c53700 10 monclient(hunting): get_auth_request con 0x7ffa1413e8e0 auth_method 0 2023-09-30T00:46:50.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.223+0000 7ffa19c53700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:50.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.223+0000 7ffa19c53700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:50.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.223+0000 7ffa19c53700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:50.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.223+0000 7ffa19452700 10 monclient(hunting): get_auth_request con 0x7ffa140a5710 auth_method 0 2023-09-30T00:46:50.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.223+0000 7ffa19452700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:50.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.223+0000 7ffa19452700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:50.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.223+0000 7ffa19452700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:50.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.223+0000 7ffa19c53700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:50.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.223+0000 7ffa19c53700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:50.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.223+0000 7ffa19c53700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:50.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.224+0000 7ffa19452700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:50.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.224+0000 7ffa19452700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:50.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.224+0000 7ffa19452700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:50.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.225+0000 7ffa19c53700 10 monclient(hunting): handle_auth_done global_id 4126 payload 274 2023-09-30T00:46:50.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.225+0000 7ffa19c53700 1 -- 172.21.15.135:0/2976029894 >> v2:172.21.15.40:3300/0 conn(0x7ffa140a5710 msgr2=0x7ffa140a75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:50.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.225+0000 7ffa19c53700 1 --2- 172.21.15.135:0/2976029894 >> v2:172.21.15.40:3300/0 conn(0x7ffa140a5710 0x7ffa140a75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.225+0000 7ffa19c53700 1 -- 172.21.15.135:0/2976029894 >> v1:172.21.15.40:6789/0 conn(0x7ffa140a7b20 legacy=0x7ffa1413e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:50.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.225+0000 7ffa19c53700 10 monclient: _finish_hunting 0 2023-09-30T00:46:50.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.225+0000 7ffa19c53700 1 monclient: found mon.noname-c 2023-09-30T00:46:50.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.225+0000 7ffa19c53700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:50.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.225+0000 7ffa19c53700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:46:50.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.225+0000 7ffa19c53700 1 -- 172.21.15.135:0/2976029894 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1413ed80 con 0x7ffa1413e8e0 2023-09-30T00:46:50.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.228+0000 7ffa19c53700 1 --2- 172.21.15.135:0/2976029894 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa1413e8e0 0x7ffa1413ae40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffa1413bf20 tx=0x7ffa0c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4265c224c3c416c1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:50.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa0bfff700 1 -- 172.21.15.135:0/2976029894 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c00e070 con 0x7ffa1413e8e0 2023-09-30T00:46:50.235 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa0bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:50.235 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa0bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:50.235 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa0bfff700 10 monclient: dump: 2023-09-30T00:46:50.235 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:50.235 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:50.235 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.235 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.235 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:50.235 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:50.235 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:50.236 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:50.236 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:50.236 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:50.236 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa0bfff700 10 monclient: _finish_auth 0 2023-09-30T00:46:50.236 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa0bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:50.236 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa0bfff700 1 -- 172.21.15.135:0/2976029894 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0c00bb60 con 0x7ffa1413e8e0 2023-09-30T00:46:50.236 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa1b6b6700 5 monclient: authenticate success, global_id 4126 2023-09-30T00:46:50.236 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa1b6b6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:50.236 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa0bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:50.236 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa1b6b6700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:50.236 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa1b6b6700 4 set_mon_vals no callback set 2023-09-30T00:46:50.237 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa0bfff700 1 -- 172.21.15.135:0/2976029894 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c012430 con 0x7ffa1413e8e0 2023-09-30T00:46:50.237 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa0bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:50.237 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa0bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:50.237 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.232+0000 7ffa0bfff700 10 monclient: dump: 2023-09-30T00:46:50.237 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:50.237 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:50.237 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.237 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.237 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:50.237 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:50.237 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:50.238 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:50.238 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:50.238 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:50.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.233+0000 7ffa1b6b6700 10 monclient: shutdown 2023-09-30T00:46:50.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.233+0000 7ffa1b6b6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:50.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.233+0000 7ffa1b6b6700 1 -- 172.21.15.135:0/2976029894 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa1413e8e0 msgr2=0x7ffa1413ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:50.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.233+0000 7ffa1b6b6700 1 --2- 172.21.15.135:0/2976029894 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa1413e8e0 0x7ffa1413ae40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffa1413bf20 tx=0x7ffa0c009780 comp rx=0 tx=0).stop 2023-09-30T00:46:50.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.233+0000 7ffa1b6b6700 1 -- 172.21.15.135:0/2976029894 shutdown_connections 2023-09-30T00:46:50.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.233+0000 7ffa1b6b6700 1 --2- 172.21.15.135:0/2976029894 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa1413e8e0 0x7ffa1413ae40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.233+0000 7ffa1b6b6700 1 --2- 172.21.15.135:0/2976029894 >> v2:172.21.15.40:3300/0 conn(0x7ffa140a5710 0x7ffa140a75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.233+0000 7ffa1b6b6700 1 -- 172.21.15.135:0/2976029894 >> 172.21.15.135:0/2976029894 conn(0x7ffa14069080 msgr2=0x7ffa140631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:50.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.233+0000 7ffa1b6b6700 1 -- 172.21.15.135:0/2976029894 shutdown_connections 2023-09-30T00:46:50.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.233+0000 7ffa1b6b6700 1 -- 172.21.15.135:0/2976029894 wait complete. 2023-09-30T00:46:50.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.233+0000 7ffa1b6b6700 10 monclient: build_initial_monmap 2023-09-30T00:46:50.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.233+0000 7ffa1b6b6700 10 monclient: monmap: 2023-09-30T00:46:50.239 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:50.239 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:50.239 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2023-09-30T00:46:50.239 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2023-09-30T00:46:50.239 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:50.239 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2023-09-30T00:46:50.239 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:46:50.240 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:46:50.240 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:46:50.240 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:50.243 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.234+0000 7ffa1b6b6700 1 librados: starting msgr at 2023-09-30T00:46:50.243 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.234+0000 7ffa1b6b6700 1 librados: starting objecter 2023-09-30T00:46:50.243 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.234+0000 7ffa1b6b6700 1 Processor -- start 2023-09-30T00:46:50.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.234+0000 7ffa1b6b6700 1 -- start start 2023-09-30T00:46:50.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.234+0000 7ffa1b6b6700 1 librados: setting wanted keys 2023-09-30T00:46:50.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.234+0000 7ffa1b6b6700 1 librados: calling monclient init 2023-09-30T00:46:50.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.234+0000 7ffa1b6b6700 10 monclient: init 2023-09-30T00:46:50.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.234+0000 7ffa1b6b6700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:50.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.234+0000 7ffa1b6b6700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:46:50.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa1b6b6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa140a5710 0x7ffa140adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:50.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa1b6b6700 10 monclient(hunting): picked mon.noname-a con 0x7ffa140a5710 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:46:50.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa1b6b6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa140a7b20 0x7ffa140a9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:50.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa1b6b6700 10 monclient(hunting): picked mon.noname-b con 0x7ffa140a7b20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:46:50.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa1b6b6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa1413e8e0 0x7ffa140ace70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:50.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa1b6b6700 10 monclient(hunting): picked mon.noname-c con 0x7ffa1413e8e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:50.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa1b6b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:50.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa1b6b6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1414e040 con 0x7ffa140a5710 2023-09-30T00:46:50.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa1b6b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:50.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa1b6b6700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140ae160 con 0x7ffa140a7b20 2023-09-30T00:46:50.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa1b6b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:50.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa1b6b6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140aa1e0 con 0x7ffa1413e8e0 2023-09-30T00:46:50.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa1b6b6700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:50.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa1b6b6700 10 monclient(hunting): authenticate will time out at 5278.674057s 2023-09-30T00:46:50.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa18c51700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa140a7b20 0x7ffa140a9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:50.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa19452700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa140a5710 0x7ffa140adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:50.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa19c53700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa1413e8e0 0x7ffa140ace70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:50.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa18c51700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa140a7b20 0x7ffa140a9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:54176/0 (socket says 172.21.15.135:54176) 2023-09-30T00:46:50.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa18c51700 1 -- 172.21.15.135:0/1301937820 learned_addr learned my addr 172.21.15.135:0/1301937820 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-09-30T00:46:50.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa18c51700 10 monclient(hunting): get_auth_request con 0x7ffa140a7b20 auth_method 0 2023-09-30T00:46:50.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa18c51700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:50.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa18c51700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:50.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa18c51700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:50.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa19452700 10 monclient(hunting): get_auth_request con 0x7ffa140a5710 auth_method 0 2023-09-30T00:46:50.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa19452700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:50.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa19452700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:50.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.235+0000 7ffa19452700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:50.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.236+0000 7ffa19c53700 10 monclient(hunting): get_auth_request con 0x7ffa1413e8e0 auth_method 0 2023-09-30T00:46:50.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.236+0000 7ffa19c53700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:50.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.236+0000 7ffa19c53700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:50.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.236+0000 7ffa19c53700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:50.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.236+0000 7ffa18c51700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:50.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.236+0000 7ffa18c51700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:50.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.236+0000 7ffa18c51700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:50.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.236+0000 7ffa19452700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:50.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.236+0000 7ffa19452700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:50.247 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.236+0000 7ffa19452700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:50.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.236+0000 7ffa19c53700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:50.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.236+0000 7ffa19c53700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:50.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.236+0000 7ffa19c53700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:50.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.239+0000 7ffa18c51700 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2023-09-30T00:46:50.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.239+0000 7ffa18c51700 1 -- 172.21.15.135:0/1301937820 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa1413e8e0 msgr2=0x7ffa140ace70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:50.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.239+0000 7ffa18c51700 1 --2- 172.21.15.135:0/1301937820 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa1413e8e0 0x7ffa140ace70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.239+0000 7ffa18c51700 1 -- 172.21.15.135:0/1301937820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa140a5710 msgr2=0x7ffa140adc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:50.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.239+0000 7ffa18c51700 1 --2- 172.21.15.135:0/1301937820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa140a5710 0x7ffa140adc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.239+0000 7ffa18c51700 10 monclient: _finish_hunting 0 2023-09-30T00:46:50.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.239+0000 7ffa18c51700 1 monclient: found mon.noname-b 2023-09-30T00:46:50.248 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.239+0000 7ffa18c51700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:50.249 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.239+0000 7ffa18c51700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:46:50.249 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.239+0000 7ffa18c51700 1 -- 172.21.15.135:0/1301937820 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c009a20 con 0x7ffa140a7b20 2023-09-30T00:46:50.249 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.242+0000 7ffa18c51700 1 --2- 172.21.15.135:0/1301937820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa140a7b20 0x7ffa140a9ca0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400c7f0 tx=0x7ffa04007f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:50.249 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.242+0000 7ffa097fa700 10 client.?.objecter ms_handle_connect 0x7ffa140a7b20 2023-09-30T00:46:50.249 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.242+0000 7ffa097fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:46:50.251 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa097fa700 1 -- 172.21.15.135:0/1301937820 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa04005b30 con 0x7ffa140a7b20 2023-09-30T00:46:50.251 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:50.251 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa097fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:50.251 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa097fa700 10 monclient: dump: 2023-09-30T00:46:50.252 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:50.252 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:50.252 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.252 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.252 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:50.252 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:50.252 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:50.252 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:50.252 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:50.252 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:50.252 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa097fa700 10 monclient: _finish_auth 0 2023-09-30T00:46:50.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa097fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:50.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa097fa700 1 -- 172.21.15.135:0/1301937820 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa04005cb0 con 0x7ffa140a7b20 2023-09-30T00:46:50.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa1b6b6700 5 monclient: authenticate success, global_id 4129 2023-09-30T00:46:50.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa1b6b6700 10 monclient: _renew_subs 2023-09-30T00:46:50.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa1b6b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:50.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa1b6b6700 1 -- 172.21.15.135:0/1301937820 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa140ad3b0 con 0x7ffa140a7b20 2023-09-30T00:46:50.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa097fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:50.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa097fa700 1 -- 172.21.15.135:0/1301937820 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa04007320 con 0x7ffa140a7b20 2023-09-30T00:46:50.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:50.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa097fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:50.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa097fa700 10 monclient: dump: 2023-09-30T00:46:50.254 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:50.254 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:50.254 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.254 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.254 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:50.254 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:50.254 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:50.254 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:50.254 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:50.254 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:50.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.246+0000 7ffa0b7fe700 4 set_mon_vals no callback set 2023-09-30T00:46:50.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.247+0000 7ffa1b6b6700 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:46:50.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.247+0000 7ffa1b6b6700 10 monclient: _renew_subs 2023-09-30T00:46:50.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.247+0000 7ffa1b6b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:50.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.247+0000 7ffa1b6b6700 1 -- 172.21.15.135:0/1301937820 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1414e4d0 con 0x7ffa140a7b20 2023-09-30T00:46:50.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.247+0000 7ffa1b6b6700 1 librados: init done 2023-09-30T00:46:50.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.247+0000 7ffa1b6b6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:50.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.247+0000 7ffa1b6b6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:50.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.247+0000 7ffa1b6b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:50.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.247+0000 7ffa1b6b6700 1 -- 172.21.15.135:0/1301937820 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1414e1f0 con 0x7ffa140a7b20 2023-09-30T00:46:50.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.250+0000 7ffa097fa700 1 -- 172.21.15.135:0/1301937820 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ffa0401c440 con 0x7ffa140a7b20 2023-09-30T00:46:50.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.250+0000 7ffa097fa700 1 -- 172.21.15.135:0/1301937820 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7ffa04005e20 con 0x7ffa140a7b20 2023-09-30T00:46:50.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.250+0000 7ffa097fa700 10 client.4129.objecter ms_dispatch 0x7ffa14001550 osd_map(4..4 src has 1..4) v4 2023-09-30T00:46:50.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.250+0000 7ffa097fa700 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2023-09-30T00:46:50.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.250+0000 7ffa097fa700 3 client.4129.objecter handle_osd_map decoding full epoch 4 2023-09-30T00:46:50.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.250+0000 7ffa097fa700 20 client.4129.objecter dump_active .. 0 homeless 2023-09-30T00:46:50.300 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:50.299+0000 7feafc14a1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T00:46:50.353 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.352+0000 7ffa097fa700 1 -- 172.21.15.135:0/1301937820 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ffa1414e1f0 con 0x7ffa140a7b20 2023-09-30T00:46:50.354 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.352+0000 7ffa097fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:50.354 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.352+0000 7ffa097fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:50.376 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:50.375+0000 7feafc14a1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T00:46:50.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.434+0000 7ffa1b6b6700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8f0de27d-3674-4ff4-8451-4fcc90786c98", "id": 2}] 2023-09-30T00:46:50.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.434+0000 7ffa1b6b6700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8f0de27d-3674-4ff4-8451-4fcc90786c98", "id": 2}] 2023-09-30T00:46:50.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.434+0000 7ffa1b6b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:50.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.434+0000 7ffa1b6b6700 1 -- 172.21.15.135:0/1301937820 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8f0de27d-3674-4ff4-8451-4fcc90786c98", "id": 2} v 0) v1 -- 0x7ffa14072b90 con 0x7ffa140a7b20 2023-09-30T00:46:50.459 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:50.458+0000 7fadf12501c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T00:46:50.475 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:50.474+0000 7feafc14a1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T00:46:50.525 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.523+0000 7ffa097fa700 1 -- 172.21.15.135:0/1301937820 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8f0de27d-3674-4ff4-8451-4fcc90786c98", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7ffa04017070 con 0x7ffa140a7b20 2023-09-30T00:46:50.525 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.523+0000 7ffa097fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8f0de27d-3674-4ff4-8451-4fcc90786c98", "id": 2}] 2023-09-30T00:46:50.525 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.523+0000 7ffa097fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:50.525 INFO:teuthology.orchestra.run.smithi135.stdout:2 2023-09-30T00:46:50.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.524+0000 7ffa1b6b6700 10 librados: watch_flush enter 2023-09-30T00:46:50.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.524+0000 7ffa1b6b6700 10 librados: watch_flush exit 2023-09-30T00:46:50.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.524+0000 7ffa1b6b6700 20 client.4129.objecter shutdown clearing up homeless session... 2023-09-30T00:46:50.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.524+0000 7ffa1b6b6700 10 client.4129.objecter successfully canceled tick 2023-09-30T00:46:50.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.525+0000 7ffa1b6b6700 10 monclient: shutdown 2023-09-30T00:46:50.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.525+0000 7ffa1b6b6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:50.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.525+0000 7ffa1b6b6700 1 -- 172.21.15.135:0/1301937820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa140a7b20 msgr2=0x7ffa140a9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:50.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.525+0000 7ffa1b6b6700 1 --2- 172.21.15.135:0/1301937820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa140a7b20 0x7ffa140a9ca0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ffa0400c7f0 tx=0x7ffa04007f30 comp rx=0 tx=0).stop 2023-09-30T00:46:50.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.525+0000 7ffa1b6b6700 1 -- 172.21.15.135:0/1301937820 shutdown_connections 2023-09-30T00:46:50.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.525+0000 7ffa1b6b6700 1 --2- 172.21.15.135:0/1301937820 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffa1413e8e0 0x7ffa140ace70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.528 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.525+0000 7ffa1b6b6700 1 --2- 172.21.15.135:0/1301937820 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa140a5710 0x7ffa140adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.528 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.525+0000 7ffa1b6b6700 1 --2- 172.21.15.135:0/1301937820 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffa140a7b20 0x7ffa140a9ca0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.528 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.525+0000 7ffa1b6b6700 1 -- 172.21.15.135:0/1301937820 >> 172.21.15.135:0/1301937820 conn(0x7ffa14069080 msgr2=0x7ffa140afb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:50.528 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.525+0000 7ffa1b6b6700 1 -- 172.21.15.135:0/1301937820 shutdown_connections 2023-09-30T00:46:50.528 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.525+0000 7ffa1b6b6700 1 -- 172.21.15.135:0/1301937820 wait complete. 2023-09-30T00:46:50.528 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.525+0000 7ffa1b6b6700 1 librados: shutdown 2023-09-30T00:46:50.541 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new d18ec2f5-7e8f-4732-a523-adc495115cbf 3 2023-09-30T00:46:50.549 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:50.548+0000 7fadf12501c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T00:46:50.660 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.658+0000 7fc0e066a700 10 monclient: get_monmap_and_config 2023-09-30T00:46:50.660 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.658+0000 7fc0e066a700 10 monclient: build_initial_monmap 2023-09-30T00:46:50.660 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.658+0000 7fc0e066a700 10 monclient: monmap: 2023-09-30T00:46:50.660 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:50.660 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:50.660 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:46:50.660575+0000 2023-09-30T00:46:50.661 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:46:50.660575+0000 2023-09-30T00:46:50.661 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:50.661 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:50.661 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:46:50.661 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:46:50.661 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:46:50.661 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:46:50.661 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:46:50.661 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:50.663 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.659+0000 7fc0e066a700 1 Processor -- start 2023-09-30T00:46:50.663 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.659+0000 7fc0e066a700 1 -- start start 2023-09-30T00:46:50.663 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.659+0000 7fc0e066a700 10 monclient: init 2023-09-30T00:46:50.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.660+0000 7fc0e066a700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:50.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.660+0000 7fc0e066a700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2023-09-30T00:46:50.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.660+0000 7fc0e066a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc0d80af2a0 0x7fc0d806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:50.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.660+0000 7fc0e066a700 10 monclient(hunting): picked mon.noname-a con 0x7fc0d80af2a0 addr v2:172.21.15.40:3300/0 2023-09-30T00:46:50.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.660+0000 7fc0e066a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0d80af670 addr v1:172.21.15.135:6789/0 2023-09-30T00:46:50.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.660+0000 7fc0e066a700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc0d80adaa0 0x7fc0d80abe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:50.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.660+0000 7fc0e066a700 10 monclient(hunting): picked mon.noname-c con 0x7fc0d80adaa0 addr v2:172.21.15.135:3300/0 2023-09-30T00:46:50.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.660+0000 7fc0e066a700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d80ad610 con 0x7fc0d80af670 2023-09-30T00:46:50.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.660+0000 7fc0e066a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:50.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.660+0000 7fc0e066a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d80ae9c0 con 0x7fc0d80af2a0 2023-09-30T00:46:50.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.660+0000 7fc0e066a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:50.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.660+0000 7fc0e066a700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d80adee0 con 0x7fc0d80adaa0 2023-09-30T00:46:50.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.660+0000 7fc0e066a700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:50.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.660+0000 7fc0e066a700 10 monclient(hunting): authenticate will time out at 5279.099455s 2023-09-30T00:46:50.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.661+0000 7fc0dec07700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc0d80adaa0 0x7fc0d80abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:50.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.661+0000 7fc0de406700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc0d80af2a0 0x7fc0d806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:50.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.661+0000 7fc0dec07700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc0d80adaa0 0x7fc0d80abe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:54184/0 (socket says 172.21.15.135:54184) 2023-09-30T00:46:50.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.661+0000 7fc0dec07700 1 -- 172.21.15.135:0/3599146878 learned_addr learned my addr 172.21.15.135:0/3599146878 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-09-30T00:46:50.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.661+0000 7fc0dec07700 10 monclient(hunting): get_auth_request con 0x7fc0d80adaa0 auth_method 0 2023-09-30T00:46:50.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.661+0000 7fc0dec07700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:50.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.661+0000 7fc0dec07700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:50.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.661+0000 7fc0dec07700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:50.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.661+0000 7fc0de406700 10 monclient(hunting): get_auth_request con 0x7fc0d80af2a0 auth_method 0 2023-09-30T00:46:50.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.661+0000 7fc0de406700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:50.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.661+0000 7fc0de406700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:50.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.661+0000 7fc0de406700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:50.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.662+0000 7fc0de406700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:50.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.662+0000 7fc0de406700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:50.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.662+0000 7fc0de406700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:50.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.662+0000 7fc0dec07700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:50.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.662+0000 7fc0dec07700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:50.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.662+0000 7fc0dec07700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:50.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.664+0000 7fc0de406700 10 monclient(hunting): handle_auth_done global_id 4131 payload 274 2023-09-30T00:46:50.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.664+0000 7fc0de406700 1 -- 172.21.15.135:0/3599146878 >> v2:172.21.15.135:3300/0 conn(0x7fc0d80adaa0 msgr2=0x7fc0d80abe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:50.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.664+0000 7fc0de406700 1 --2- 172.21.15.135:0/3599146878 >> v2:172.21.15.135:3300/0 conn(0x7fc0d80adaa0 0x7fc0d80abe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.664+0000 7fc0de406700 1 -- 172.21.15.135:0/3599146878 >> v1:172.21.15.135:6789/0 conn(0x7fc0d80af670 legacy=0x7fc0d813c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:50.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.664+0000 7fc0de406700 10 monclient: _finish_hunting 0 2023-09-30T00:46:50.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.664+0000 7fc0de406700 1 monclient: found mon.noname-a 2023-09-30T00:46:50.668 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.664+0000 7fc0de406700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:50.668 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.664+0000 7fc0de406700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:46:50.668 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.664+0000 7fc0de406700 1 -- 172.21.15.135:0/3599146878 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d80aebc0 con 0x7fc0d80af2a0 2023-09-30T00:46:50.668 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:50.666+0000 7feafc14a1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T00:46:50.668 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.667+0000 7fc0de406700 1 --2- 172.21.15.135:0/3599146878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d80af2a0 0x7fc0d806ee00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc0d806f370 tx=0x7fc0c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6847f3807668d7db server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:50.673 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0dd404700 1 -- 172.21.15.135:0/3599146878 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8011630 con 0x7fc0d80af2a0 2023-09-30T00:46:50.674 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0dd404700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:50.674 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0dd404700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:50.674 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0dd404700 10 monclient: dump: 2023-09-30T00:46:50.674 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:50.674 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:50.674 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.674 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.674 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:50.674 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:50.674 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:50.674 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:50.675 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:50.675 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:50.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0dd404700 10 monclient: _finish_auth 0 2023-09-30T00:46:50.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0dd404700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:50.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0dd404700 1 -- 172.21.15.135:0/3599146878 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c8011790 con 0x7fc0d80af2a0 2023-09-30T00:46:50.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0dd404700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:50.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0dd404700 1 -- 172.21.15.135:0/3599146878 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8011b30 con 0x7fc0d80af2a0 2023-09-30T00:46:50.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0e066a700 5 monclient: authenticate success, global_id 4131 2023-09-30T00:46:50.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0e066a700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:50.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0e066a700 4 set_mon_vals no callback set 2023-09-30T00:46:50.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0dd404700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:50.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0dd404700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:50.676 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.672+0000 7fc0dd404700 10 monclient: dump: 2023-09-30T00:46:50.676 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:50.676 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:50.676 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.676 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.676 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:50.676 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:50.676 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:50.676 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:50.676 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:50.676 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:50.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.673+0000 7fc0e066a700 10 monclient: shutdown 2023-09-30T00:46:50.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.673+0000 7fc0e066a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:50.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.673+0000 7fc0e066a700 1 -- 172.21.15.135:0/3599146878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d80af2a0 msgr2=0x7fc0d806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:50.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.673+0000 7fc0e066a700 1 --2- 172.21.15.135:0/3599146878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d80af2a0 0x7fc0d806ee00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc0d806f370 tx=0x7fc0c80091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.673+0000 7fc0e066a700 1 -- 172.21.15.135:0/3599146878 shutdown_connections 2023-09-30T00:46:50.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.673+0000 7fc0e066a700 1 --2- 172.21.15.135:0/3599146878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d80af2a0 0x7fc0d806ee00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.673+0000 7fc0e066a700 1 --2- 172.21.15.135:0/3599146878 >> v2:172.21.15.135:3300/0 conn(0x7fc0d80adaa0 0x7fc0d80abe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.673+0000 7fc0e066a700 1 -- 172.21.15.135:0/3599146878 >> 172.21.15.135:0/3599146878 conn(0x7fc0d806af20 msgr2=0x7fc0d80631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:50.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.673+0000 7fc0e066a700 1 -- 172.21.15.135:0/3599146878 shutdown_connections 2023-09-30T00:46:50.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.673+0000 7fc0e066a700 1 -- 172.21.15.135:0/3599146878 wait complete. 2023-09-30T00:46:50.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.674+0000 7fc0e066a700 10 monclient: build_initial_monmap 2023-09-30T00:46:50.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.674+0000 7fc0e066a700 10 monclient: monmap: 2023-09-30T00:46:50.681 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:50.681 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:50.681 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2023-09-30T00:46:50.681 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2023-09-30T00:46:50.681 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:50.681 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2023-09-30T00:46:50.681 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:46:50.682 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:46:50.682 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:46:50.682 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:50.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.675+0000 7fc0e066a700 1 librados: starting msgr at 2023-09-30T00:46:50.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.675+0000 7fc0e066a700 1 librados: starting objecter 2023-09-30T00:46:50.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.675+0000 7fc0e066a700 1 Processor -- start 2023-09-30T00:46:50.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.675+0000 7fc0e066a700 1 -- start start 2023-09-30T00:46:50.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.675+0000 7fc0e066a700 1 librados: setting wanted keys 2023-09-30T00:46:50.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.675+0000 7fc0e066a700 1 librados: calling monclient init 2023-09-30T00:46:50.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.675+0000 7fc0e066a700 10 monclient: init 2023-09-30T00:46:50.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:50.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:46:50.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0d80adaa0 0x7fc0d813fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:50.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 10 monclient(hunting): picked mon.noname-b con 0x7fc0d80adaa0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:46:50.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d80af2a0 0x7fc0d814c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:50.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 10 monclient(hunting): picked mon.noname-a con 0x7fc0d80af2a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:46:50.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0d80af670 0x7fc0d814c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:50.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 10 monclient(hunting): picked mon.noname-c con 0x7fc0d80af670 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:50.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:50.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d814cdf0 con 0x7fc0d80af2a0 2023-09-30T00:46:50.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:50.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d814be90 con 0x7fc0d80adaa0 2023-09-30T00:46:50.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:50.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d8140390 con 0x7fc0d80af670 2023-09-30T00:46:50.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:50.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.676+0000 7fc0e066a700 10 monclient(hunting): authenticate will time out at 5279.115484s 2023-09-30T00:46:50.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0de406700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0d80adaa0 0x7fc0d813fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:50.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0ddc05700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d80af2a0 0x7fc0d814c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:50.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0dec07700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0d80af670 0x7fc0d814c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:50.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0de406700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0d80adaa0 0x7fc0d813fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:54188/0 (socket says 172.21.15.135:54188) 2023-09-30T00:46:50.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0de406700 1 -- 172.21.15.135:0/1362346325 learned_addr learned my addr 172.21.15.135:0/1362346325 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-09-30T00:46:50.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0de406700 10 monclient(hunting): get_auth_request con 0x7fc0d80adaa0 auth_method 0 2023-09-30T00:46:50.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0de406700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:50.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0de406700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:50.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0de406700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:50.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0ddc05700 10 monclient(hunting): get_auth_request con 0x7fc0d80af2a0 auth_method 0 2023-09-30T00:46:50.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0ddc05700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:50.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0ddc05700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:50.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0ddc05700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:50.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0dec07700 10 monclient(hunting): get_auth_request con 0x7fc0d80af670 auth_method 0 2023-09-30T00:46:50.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0dec07700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:50.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0dec07700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:50.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0dec07700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:50.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0de406700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:50.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0de406700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:50.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.677+0000 7fc0de406700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:50.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.678+0000 7fc0ddc05700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:50.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.678+0000 7fc0ddc05700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:50.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.678+0000 7fc0ddc05700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:50.686 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.678+0000 7fc0dec07700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:50.686 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.678+0000 7fc0dec07700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:50.686 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.678+0000 7fc0dec07700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:50.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.681+0000 7fc0de406700 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2023-09-30T00:46:50.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.681+0000 7fc0de406700 1 -- 172.21.15.135:0/1362346325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0d80af670 msgr2=0x7fc0d814c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:50.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.681+0000 7fc0de406700 1 --2- 172.21.15.135:0/1362346325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0d80af670 0x7fc0d814c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.681+0000 7fc0de406700 1 -- 172.21.15.135:0/1362346325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d80af2a0 msgr2=0x7fc0d814c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:50.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.681+0000 7fc0de406700 1 --2- 172.21.15.135:0/1362346325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d80af2a0 0x7fc0d814c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.681+0000 7fc0de406700 10 monclient: _finish_hunting 0 2023-09-30T00:46:50.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.681+0000 7fc0de406700 1 monclient: found mon.noname-b 2023-09-30T00:46:50.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.681+0000 7fc0de406700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:50.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.681+0000 7fc0de406700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:46:50.688 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.681+0000 7fc0de406700 1 -- 172.21.15.135:0/1362346325 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c8009420 con 0x7fc0d80adaa0 2023-09-30T00:46:50.688 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.683+0000 7fc0de406700 1 --2- 172.21.15.135:0/1362346325 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0d80adaa0 0x7fc0d813fa80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8063fc0 tx=0x7fc0c8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:50.688 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.683+0000 7fc0ce7fc700 10 client.?.objecter ms_handle_connect 0x7fc0d80adaa0 2023-09-30T00:46:50.688 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.683+0000 7fc0ce7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:46:50.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0ce7fc700 1 -- 172.21.15.135:0/1362346325 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8000d10 con 0x7fc0d80adaa0 2023-09-30T00:46:50.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:50.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0ce7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:50.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0ce7fc700 10 monclient: dump: 2023-09-30T00:46:50.689 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:50.690 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:50.690 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.690 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.690 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:50.690 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:50.690 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:50.690 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:50.690 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:50.690 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:50.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0ce7fc700 10 monclient: _finish_auth 0 2023-09-30T00:46:50.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0ce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:50.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0e066a700 5 monclient: authenticate success, global_id 4135 2023-09-30T00:46:50.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0ce7fc700 1 -- 172.21.15.135:0/1362346325 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c8011b10 con 0x7fc0d80adaa0 2023-09-30T00:46:50.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0e066a700 10 monclient: _renew_subs 2023-09-30T00:46:50.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0e066a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:50.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0e066a700 1 -- 172.21.15.135:0/1362346325 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d813ec60 con 0x7fc0d80adaa0 2023-09-30T00:46:50.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0ce7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:50.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0ce7fc700 1 -- 172.21.15.135:0/1362346325 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c8025620 con 0x7fc0d80adaa0 2023-09-30T00:46:50.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:50.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0ce7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:50.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0ce7fc700 10 monclient: dump: 2023-09-30T00:46:50.691 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:50.691 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:50.692 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.692 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:50.692 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:50.692 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:50.692 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:50.692 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:50.692 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:50.692 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:50.692 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0e066a700 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:46:50.692 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0e066a700 10 monclient: _renew_subs 2023-09-30T00:46:50.692 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0e066a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:50.692 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0e066a700 1 -- 172.21.15.135:0/1362346325 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d8140060 con 0x7fc0d80adaa0 2023-09-30T00:46:50.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0e066a700 1 librados: init done 2023-09-30T00:46:50.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.687+0000 7fc0dcc03700 4 set_mon_vals no callback set 2023-09-30T00:46:50.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.688+0000 7fc0e066a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:50.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.688+0000 7fc0e066a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:50.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.688+0000 7fc0e066a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:50.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.688+0000 7fc0e066a700 1 -- 172.21.15.135:0/1362346325 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d8064a00 con 0x7fc0d80adaa0 2023-09-30T00:46:50.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.692+0000 7fc0ce7fc700 1 -- 172.21.15.135:0/1362346325 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc0c8025c50 con 0x7fc0d80adaa0 2023-09-30T00:46:50.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.692+0000 7fc0ce7fc700 1 -- 172.21.15.135:0/1362346325 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7fc0c8019850 con 0x7fc0d80adaa0 2023-09-30T00:46:50.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.692+0000 7fc0ce7fc700 10 client.4135.objecter ms_dispatch 0x7fc0d8001550 osd_map(5..5 src has 1..5) v4 2023-09-30T00:46:50.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.692+0000 7fc0ce7fc700 3 client.4135.objecter handle_osd_map got epochs [5,5] > 0 2023-09-30T00:46:50.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.692+0000 7fc0ce7fc700 3 client.4135.objecter handle_osd_map decoding full epoch 5 2023-09-30T00:46:50.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.692+0000 7fc0ce7fc700 20 client.4135.objecter dump_active .. 0 homeless 2023-09-30T00:46:50.705 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:50.704+0000 7fadf12501c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T00:46:50.753 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:50.752+0000 7feafc14a1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T00:46:50.781 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:50.780+0000 7fadf12501c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T00:46:50.798 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.796+0000 7fc0ce7fc700 1 -- 172.21.15.135:0/1362346325 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fc0c8033060 con 0x7fc0d80adaa0 2023-09-30T00:46:50.798 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.796+0000 7fc0ce7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:50.798 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.796+0000 7fc0ce7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:50.880 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.879+0000 7fc0e066a700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d18ec2f5-7e8f-4732-a523-adc495115cbf", "id": 3}] 2023-09-30T00:46:50.881 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.879+0000 7fc0e066a700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d18ec2f5-7e8f-4732-a523-adc495115cbf", "id": 3}] 2023-09-30T00:46:50.881 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.879+0000 7fc0e066a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:50.881 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.879+0000 7fc0e066a700 1 -- 172.21.15.135:0/1362346325 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d18ec2f5-7e8f-4732-a523-adc495115cbf", "id": 3} v 0) v1 -- 0x7fc0d80ad210 con 0x7fc0d80adaa0 2023-09-30T00:46:50.976 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.975+0000 7fc0ce7fc700 1 -- 172.21.15.135:0/1362346325 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d18ec2f5-7e8f-4732-a523-adc495115cbf", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc0c8015020 con 0x7fc0d80adaa0 2023-09-30T00:46:50.977 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.975+0000 7fc0ce7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d18ec2f5-7e8f-4732-a523-adc495115cbf", "id": 3}] 2023-09-30T00:46:50.977 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.975+0000 7fc0ce7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:50.977 INFO:teuthology.orchestra.run.smithi135.stdout:3 2023-09-30T00:46:50.978 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 10 librados: watch_flush enter 2023-09-30T00:46:50.978 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 10 librados: watch_flush exit 2023-09-30T00:46:50.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 20 client.4135.objecter shutdown clearing up homeless session... 2023-09-30T00:46:50.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 10 client.4135.objecter successfully canceled tick 2023-09-30T00:46:50.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 10 monclient: shutdown 2023-09-30T00:46:50.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:50.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 1 -- 172.21.15.135:0/1362346325 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0d80adaa0 msgr2=0x7fc0d813fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:50.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 1 --2- 172.21.15.135:0/1362346325 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0d80adaa0 0x7fc0d813fa80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8063fc0 tx=0x7fc0c8017000 comp rx=0 tx=0).stop 2023-09-30T00:46:50.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 1 -- 172.21.15.135:0/1362346325 shutdown_connections 2023-09-30T00:46:50.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 1 --2- 172.21.15.135:0/1362346325 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0d80adaa0 0x7fc0d813fa80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 1 --2- 172.21.15.135:0/1362346325 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc0d80af670 0x7fc0d814c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 1 --2- 172.21.15.135:0/1362346325 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0d80af2a0 0x7fc0d814c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:50.980 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 1 -- 172.21.15.135:0/1362346325 >> 172.21.15.135:0/1362346325 conn(0x7fc0d806af20 msgr2=0x7fc0d80631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:50.980 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 1 -- 172.21.15.135:0/1362346325 shutdown_connections 2023-09-30T00:46:50.980 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 1 -- 172.21.15.135:0/1362346325 wait complete. 2023-09-30T00:46:50.980 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:50.976+0000 7fc0e066a700 1 librados: shutdown 2023-09-30T00:46:50.989 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 4e2465fe-bb27-47bd-8ddf-f334cac6041a 4 2023-09-30T00:46:50.992 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:50.991+0000 7fadf12501c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T00:46:51.053 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:51.052+0000 7feafc14a1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T00:46:51.078 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:51.077+0000 7fadf12501c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T00:46:51.107 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.105+0000 7f2a2feae700 10 monclient: get_monmap_and_config 2023-09-30T00:46:51.107 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.105+0000 7f2a2feae700 10 monclient: build_initial_monmap 2023-09-30T00:46:51.107 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.105+0000 7f2a2feae700 10 monclient: monmap: 2023-09-30T00:46:51.107 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:51.107 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:51.107 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:46:51.106896+0000 2023-09-30T00:46:51.107 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:46:51.106896+0000 2023-09-30T00:46:51.107 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:51.107 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:51.108 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:46:51.108 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:46:51.108 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:46:51.108 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:46:51.108 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:46:51.108 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:51.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.105+0000 7f2a2feae700 1 Processor -- start 2023-09-30T00:46:51.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.106+0000 7f2a2feae700 1 -- start start 2023-09-30T00:46:51.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.106+0000 7f2a2feae700 10 monclient: init 2023-09-30T00:46:51.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.106+0000 7f2a2feae700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:51.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.106+0000 7f2a2feae700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-09-30T00:46:51.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.106+0000 7f2a2feae700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a280b1460 0x7f2a280b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:51.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.106+0000 7f2a2feae700 10 monclient(hunting): picked mon.noname-b con 0x7f2a280b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:51.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.106+0000 7f2a2feae700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a280a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:46:51.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.106+0000 7f2a2feae700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a2813ac20 addr v1:172.21.15.40:6789/0 2023-09-30T00:46:51.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.106+0000 7f2a2feae700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a280a5b20 con 0x7f2a2813ac20 2023-09-30T00:46:51.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.106+0000 7f2a2feae700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a280a71f0 con 0x7f2a280a5fb0 2023-09-30T00:46:51.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.106+0000 7f2a2feae700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:51.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.106+0000 7f2a2feae700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a280a74d0 con 0x7f2a280b1460 2023-09-30T00:46:51.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.106+0000 7f2a2feae700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:51.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.106+0000 7f2a2feae700 10 monclient(hunting): authenticate will time out at 5279.545798s 2023-09-30T00:46:51.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.107+0000 7f2a2d449700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f2a280a5fb0 0x7f2a28141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:54816/0 (socket says 172.21.15.135:54816) 2023-09-30T00:46:51.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.107+0000 7f2a2d449700 1 -- 172.21.15.135:0/2948211003 learned_addr learned my addr 172.21.15.135:0/2948211003 (peer_addr_for_me v1:172.21.15.135:0/0) 2023-09-30T00:46:51.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.107+0000 7f2a2dc4a700 1 --2- 172.21.15.135:0/2948211003 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a280b1460 0x7f2a280b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:51.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.108+0000 7f2a2dc4a700 10 monclient(hunting): get_auth_request con 0x7f2a280b1460 auth_method 0 2023-09-30T00:46:51.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.108+0000 7f2a2dc4a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:51.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.108+0000 7f2a2dc4a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:51.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.108+0000 7f2a2dc4a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:51.112 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.108+0000 7f2a2dc4a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:51.112 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.108+0000 7f2a2dc4a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:51.112 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.108+0000 7f2a2dc4a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:51.113 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.110+0000 7f2a2cc48700 1 -- 172.21.15.135:0/2948211003 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617480969 0 0) 0x7f2a280a71f0 con 0x7f2a280a5fb0 2023-09-30T00:46:51.113 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.110+0000 7f2a2cc48700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:51.113 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.110+0000 7f2a2cc48700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:51.113 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.110+0000 7f2a2cc48700 10 monclient(hunting): my global_id is 4138 2023-09-30T00:46:51.113 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.110+0000 7f2a2cc48700 1 -- 172.21.15.135:0/2948211003 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a14003570 con 0x7f2a280a5fb0 2023-09-30T00:46:51.113 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.111+0000 7f2a2dc4a700 10 monclient(hunting): handle_auth_done global_id 4121 payload 274 2023-09-30T00:46:51.113 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.111+0000 7f2a2dc4a700 1 -- 172.21.15.135:0/2948211003 >> v1:172.21.15.135:6789/0 conn(0x7f2a280a5fb0 legacy=0x7f2a28141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:51.113 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.111+0000 7f2a2dc4a700 1 -- 172.21.15.135:0/2948211003 >> v1:172.21.15.40:6789/0 conn(0x7f2a2813ac20 legacy=0x7f2a2813b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:51.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.111+0000 7f2a2dc4a700 10 monclient: _finish_hunting 0 2023-09-30T00:46:51.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.111+0000 7f2a2dc4a700 1 monclient: found mon.noname-b 2023-09-30T00:46:51.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.111+0000 7f2a2dc4a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:51.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.111+0000 7f2a2dc4a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:46:51.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.111+0000 7f2a2dc4a700 1 -- 172.21.15.135:0/2948211003 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a280a63c0 con 0x7f2a280b1460 2023-09-30T00:46:51.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.113+0000 7f2a2dc4a700 1 --2- 172.21.15.135:0/2948211003 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a280b1460 0x7f2a280b1830 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009910 tx=0x7f2a240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76f6ebedbe1cbf12 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:51.119 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2cc48700 1 -- 172.21.15.135:0/2948211003 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2400b680 con 0x7f2a280b1460 2023-09-30T00:46:51.119 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2cc48700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:51.120 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2cc48700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:51.120 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2cc48700 10 monclient: dump: 2023-09-30T00:46:51.120 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:51.120 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:51.120 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.120 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.120 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:51.120 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:51.120 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:51.120 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:51.120 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:51.121 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:51.121 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2cc48700 10 monclient: _finish_auth 0 2023-09-30T00:46:51.121 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2cc48700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:51.121 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2cc48700 1 -- 172.21.15.135:0/2948211003 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2400b7e0 con 0x7f2a280b1460 2023-09-30T00:46:51.121 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2cc48700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:51.121 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2cc48700 1 -- 172.21.15.135:0/2948211003 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2400bba0 con 0x7f2a280b1460 2023-09-30T00:46:51.121 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2cc48700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:51.121 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2cc48700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:51.121 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2cc48700 10 monclient: dump: 2023-09-30T00:46:51.121 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:51.121 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:51.122 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.122 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.122 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:51.122 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:51.122 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:51.122 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:51.122 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:51.122 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:51.122 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2feae700 5 monclient: authenticate success, global_id 4121 2023-09-30T00:46:51.122 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2feae700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:51.122 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.118+0000 7f2a2feae700 4 set_mon_vals no callback set 2023-09-30T00:46:51.126 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.119+0000 7f2a2feae700 10 monclient: shutdown 2023-09-30T00:46:51.126 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.119+0000 7f2a2feae700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:51.127 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.119+0000 7f2a2feae700 1 -- 172.21.15.135:0/2948211003 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a280b1460 msgr2=0x7f2a280b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:51.127 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.119+0000 7f2a2feae700 1 --2- 172.21.15.135:0/2948211003 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a280b1460 0x7f2a280b1830 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009910 tx=0x7f2a240091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.127 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.119+0000 7f2a2feae700 1 -- 172.21.15.135:0/2948211003 shutdown_connections 2023-09-30T00:46:51.127 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.119+0000 7f2a2feae700 1 --2- 172.21.15.135:0/2948211003 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a280b1460 0x7f2a280b1830 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.127 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.119+0000 7f2a2feae700 1 -- 172.21.15.135:0/2948211003 >> 172.21.15.135:0/2948211003 conn(0x7f2a280631f0 msgr2=0x7f2a280635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:51.127 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.119+0000 7f2a2feae700 1 -- 172.21.15.135:0/2948211003 shutdown_connections 2023-09-30T00:46:51.127 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.119+0000 7f2a2feae700 1 -- 172.21.15.135:0/2948211003 wait complete. 2023-09-30T00:46:51.127 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.120+0000 7f2a2feae700 10 monclient: build_initial_monmap 2023-09-30T00:46:51.127 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.120+0000 7f2a2feae700 10 monclient: monmap: 2023-09-30T00:46:51.127 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:51.128 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:51.128 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2023-09-30T00:46:51.128 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2023-09-30T00:46:51.128 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:51.128 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2023-09-30T00:46:51.128 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:46:51.128 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:46:51.128 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:46:51.128 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:51.128 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.120+0000 7f2a2feae700 1 librados: starting msgr at 2023-09-30T00:46:51.128 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.120+0000 7f2a2feae700 1 librados: starting objecter 2023-09-30T00:46:51.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.120+0000 7f2a2feae700 1 Processor -- start 2023-09-30T00:46:51.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.120+0000 7f2a2feae700 1 -- start start 2023-09-30T00:46:51.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.120+0000 7f2a2feae700 1 librados: setting wanted keys 2023-09-30T00:46:51.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.120+0000 7f2a2feae700 1 librados: calling monclient init 2023-09-30T00:46:51.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.120+0000 7f2a2feae700 10 monclient: init 2023-09-30T00:46:51.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.120+0000 7f2a2feae700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:51.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.120+0000 7f2a2feae700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:46:51.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.120+0000 7f2a2feae700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a280a5fb0 0x7f2a2813e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:51.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.120+0000 7f2a2feae700 10 monclient(hunting): picked mon.noname-a con 0x7f2a280a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:46:51.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2feae700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a280b1460 0x7f2a280a79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:51.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2feae700 10 monclient(hunting): picked mon.noname-b con 0x7f2a280b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:46:51.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2feae700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a2813ac20 0x7f2a280a7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:51.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2feae700 10 monclient(hunting): picked mon.noname-c con 0x7f2a2813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:51.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2feae700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:51.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2feae700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a280aa040 con 0x7f2a280a5fb0 2023-09-30T00:46:51.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2feae700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:51.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2feae700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a280ab8b0 con 0x7f2a280b1460 2023-09-30T00:46:51.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2feae700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:51.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2feae700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a280abb90 con 0x7f2a2813ac20 2023-09-30T00:46:51.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2feae700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:51.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2feae700 10 monclient(hunting): authenticate will time out at 5279.560046s 2023-09-30T00:46:51.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2d449700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a280b1460 0x7f2a280a79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:51.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2e44b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a2813ac20 0x7f2a280a7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:51.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2dc4a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a280a5fb0 0x7f2a2813e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:51.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2d449700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a280b1460 0x7f2a280a79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:54190/0 (socket says 172.21.15.135:54190) 2023-09-30T00:46:51.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2d449700 1 -- 172.21.15.135:0/3288913765 learned_addr learned my addr 172.21.15.135:0/3288913765 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-09-30T00:46:51.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2d449700 10 monclient(hunting): get_auth_request con 0x7f2a280b1460 auth_method 0 2023-09-30T00:46:51.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2d449700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:51.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.121+0000 7f2a2d449700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:51.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2d449700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:51.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2e44b700 10 monclient(hunting): get_auth_request con 0x7f2a2813ac20 auth_method 0 2023-09-30T00:46:51.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2e44b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:51.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2e44b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:51.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2e44b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:51.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2dc4a700 10 monclient(hunting): get_auth_request con 0x7f2a280a5fb0 auth_method 0 2023-09-30T00:46:51.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2dc4a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:51.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2dc4a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:51.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2dc4a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:51.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2d449700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:51.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2d449700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:51.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2d449700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:51.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2e44b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:51.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2e44b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:51.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.122+0000 7f2a2e44b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:51.133 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.123+0000 7f2a2dc4a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:51.133 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.123+0000 7f2a2dc4a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:51.133 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.123+0000 7f2a2dc4a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:51.133 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:51.125+0000 7feafc14a1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T00:46:51.135 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.126+0000 7f2a2d449700 10 monclient(hunting): handle_auth_done global_id 4141 payload 824 2023-09-30T00:46:51.135 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.126+0000 7f2a2d449700 1 -- 172.21.15.135:0/3288913765 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a2813ac20 msgr2=0x7f2a280a7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:51.135 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.126+0000 7f2a2d449700 1 --2- 172.21.15.135:0/3288913765 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a2813ac20 0x7f2a280a7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.135 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.126+0000 7f2a2d449700 1 -- 172.21.15.135:0/3288913765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a280a5fb0 msgr2=0x7f2a2813e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:51.135 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.126+0000 7f2a2d449700 1 --2- 172.21.15.135:0/3288913765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a280a5fb0 0x7f2a2813e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.135 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.126+0000 7f2a2d449700 10 monclient: _finish_hunting 0 2023-09-30T00:46:51.136 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.126+0000 7f2a2d449700 1 monclient: found mon.noname-b 2023-09-30T00:46:51.136 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.126+0000 7f2a2d449700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:51.136 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.126+0000 7f2a2d449700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:46:51.136 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.126+0000 7f2a2d449700 1 -- 172.21.15.135:0/3288913765 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a24009420 con 0x7f2a280b1460 2023-09-30T00:46:51.136 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.129+0000 7f2a2d449700 1 --2- 172.21.15.135:0/3288913765 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a280b1460 0x7f2a280a79f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800c7c0 tx=0x7f2a18007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:51.136 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.129+0000 7f2a1dffb700 10 client.?.objecter ms_handle_connect 0x7f2a280b1460 2023-09-30T00:46:51.136 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.129+0000 7f2a1dffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:46:51.136 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a1dffb700 1 -- 172.21.15.135:0/3288913765 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18005cd0 con 0x7f2a280b1460 2023-09-30T00:46:51.136 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a1dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:51.136 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a1dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:51.136 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a1dffb700 10 monclient: dump: 2023-09-30T00:46:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:51.137 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a1dffb700 10 monclient: _finish_auth 0 2023-09-30T00:46:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a1dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a1dffb700 1 -- 172.21.15.135:0/3288913765 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a18005e30 con 0x7f2a280b1460 2023-09-30T00:46:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a2feae700 5 monclient: authenticate success, global_id 4141 2023-09-30T00:46:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a1dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a1dffb700 1 -- 172.21.15.135:0/3288913765 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18007320 con 0x7f2a280b1460 2023-09-30T00:46:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a1dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a1dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a1dffb700 10 monclient: dump: 2023-09-30T00:46:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:51.139 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a1ffff700 4 set_mon_vals no callback set 2023-09-30T00:46:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a2feae700 10 monclient: _renew_subs 2023-09-30T00:46:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a2feae700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a2feae700 1 -- 172.21.15.135:0/3288913765 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a280aa300 con 0x7f2a280b1460 2023-09-30T00:46:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a2feae700 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:46:51.140 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a2feae700 10 monclient: _renew_subs 2023-09-30T00:46:51.140 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a2feae700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:51.140 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a2feae700 1 -- 172.21.15.135:0/3288913765 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a280a9c80 con 0x7f2a280b1460 2023-09-30T00:46:51.140 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.133+0000 7f2a2feae700 1 librados: init done 2023-09-30T00:46:51.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.134+0000 7f2a2feae700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:51.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.134+0000 7f2a2feae700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:51.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.134+0000 7f2a2feae700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:51.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.134+0000 7f2a2feae700 1 -- 172.21.15.135:0/3288913765 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a280aa520 con 0x7f2a280b1460 2023-09-30T00:46:51.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.138+0000 7f2a1dffb700 1 -- 172.21.15.135:0/3288913765 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f2a1801c6c0 con 0x7f2a280b1460 2023-09-30T00:46:51.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.138+0000 7f2a1dffb700 1 -- 172.21.15.135:0/3288913765 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f2a18007710 con 0x7f2a280b1460 2023-09-30T00:46:51.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.138+0000 7f2a1dffb700 10 client.4141.objecter ms_dispatch 0x7f2a28001550 osd_map(6..6 src has 1..6) v4 2023-09-30T00:46:51.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.138+0000 7f2a1dffb700 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2023-09-30T00:46:51.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.138+0000 7f2a1dffb700 3 client.4141.objecter handle_osd_map decoding full epoch 6 2023-09-30T00:46:51.142 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.138+0000 7f2a1dffb700 20 client.4141.objecter dump_active .. 0 homeless 2023-09-30T00:46:51.163 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:51.162+0000 7fadf12501c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T00:46:51.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.238+0000 7f2a1dffb700 1 -- 172.21.15.135:0/3288913765 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f2a18017070 con 0x7f2a280b1460 2023-09-30T00:46:51.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.238+0000 7f2a1dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:51.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.238+0000 7f2a1dffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:51.275 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:51.274+0000 7fadf12501c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T00:46:51.325 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.323+0000 7f2a2feae700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4e2465fe-bb27-47bd-8ddf-f334cac6041a", "id": 4}] 2023-09-30T00:46:51.325 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.323+0000 7f2a2feae700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4e2465fe-bb27-47bd-8ddf-f334cac6041a", "id": 4}] 2023-09-30T00:46:51.325 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.323+0000 7f2a2feae700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:51.325 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.323+0000 7f2a2feae700 1 -- 172.21.15.135:0/3288913765 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4e2465fe-bb27-47bd-8ddf-f334cac6041a", "id": 4} v 0) v1 -- 0x7f2a28064a00 con 0x7f2a280b1460 2023-09-30T00:46:51.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.416+0000 7f2a1dffb700 1 -- 172.21.15.135:0/3288913765 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4e2465fe-bb27-47bd-8ddf-f334cac6041a", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2a1801ca80 con 0x7f2a280b1460 2023-09-30T00:46:51.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.416+0000 7f2a1dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4e2465fe-bb27-47bd-8ddf-f334cac6041a", "id": 4}] 2023-09-30T00:46:51.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.416+0000 7f2a1dffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:51.418 INFO:teuthology.orchestra.run.smithi135.stdout:4 2023-09-30T00:46:51.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 10 librados: watch_flush enter 2023-09-30T00:46:51.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 10 librados: watch_flush exit 2023-09-30T00:46:51.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 20 client.4141.objecter shutdown clearing up homeless session... 2023-09-30T00:46:51.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 10 client.4141.objecter successfully canceled tick 2023-09-30T00:46:51.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 10 monclient: shutdown 2023-09-30T00:46:51.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:51.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 1 -- 172.21.15.135:0/3288913765 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a280b1460 msgr2=0x7f2a280a79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:51.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 1 --2- 172.21.15.135:0/3288913765 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a280b1460 0x7f2a280a79f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800c7c0 tx=0x7f2a18007e00 comp rx=0 tx=0).stop 2023-09-30T00:46:51.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 1 -- 172.21.15.135:0/3288913765 shutdown_connections 2023-09-30T00:46:51.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 1 --2- 172.21.15.135:0/3288913765 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a2813ac20 0x7f2a280a7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 1 --2- 172.21.15.135:0/3288913765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a280a5fb0 0x7f2a2813e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 1 --2- 172.21.15.135:0/3288913765 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a280b1460 0x7f2a280a79f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 1 -- 172.21.15.135:0/3288913765 >> 172.21.15.135:0/3288913765 conn(0x7f2a280631f0 msgr2=0x7f2a280b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:51.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 1 -- 172.21.15.135:0/3288913765 shutdown_connections 2023-09-30T00:46:51.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 1 -- 172.21.15.135:0/3288913765 wait complete. 2023-09-30T00:46:51.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.417+0000 7f2a2feae700 1 librados: shutdown 2023-09-30T00:46:51.430 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 7ea449db-9b22-48b8-a61b-1937eeeb6c8e 5 2023-09-30T00:46:51.499 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:51.498+0000 7fadf12501c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T00:46:51.548 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.547+0000 7fd5d6944700 10 monclient: get_monmap_and_config 2023-09-30T00:46:51.548 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.547+0000 7fd5d6944700 10 monclient: build_initial_monmap 2023-09-30T00:46:51.548 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.547+0000 7fd5d6944700 10 monclient: monmap: 2023-09-30T00:46:51.548 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:51.548 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:51.549 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:46:51.548855+0000 2023-09-30T00:46:51.549 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:46:51.548855+0000 2023-09-30T00:46:51.549 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:51.549 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:51.549 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:46:51.549 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:46:51.549 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:46:51.549 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:46:51.549 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:46:51.549 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:51.552 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.547+0000 7fd5d6944700 1 Processor -- start 2023-09-30T00:46:51.552 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 1 -- start start 2023-09-30T00:46:51.552 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 10 monclient: init 2023-09-30T00:46:51.552 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:51.552 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-09-30T00:46:51.552 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5d00b1460 0x7fd5d00b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:51.552 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 10 monclient(hunting): picked mon.noname-b con 0x7fd5d00b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:51.552 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd5d00a5fb0 0x7fd5d0141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:51.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 10 monclient(hunting): picked mon.noname-c con 0x7fd5d00a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:46:51.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5d00af410 addr v1:172.21.15.40:6789/0 2023-09-30T00:46:51.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d00a71f0 con 0x7fd5d00af410 2023-09-30T00:46:51.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:51.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5d00a74d0 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:51.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00a63c0 con 0x7fd5d00b1460 2023-09-30T00:46:51.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:51.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.548+0000 7fd5d6944700 10 monclient(hunting): authenticate will time out at 5279.987753s 2023-09-30T00:46:51.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.549+0000 7fd5d4ee1700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd5d00af410 0x7fd5d00af890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.135:51924/0 (socket says 172.21.15.135:51924) 2023-09-30T00:46:51.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.549+0000 7fd5d4ee1700 1 -- 172.21.15.135:0/1365900838 learned_addr learned my addr 172.21.15.135:0/1365900838 (peer_addr_for_me v1:172.21.15.135:0/0) 2023-09-30T00:46:51.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.549+0000 7fd5cf7fe700 1 --2- 172.21.15.135:0/1365900838 >> v2:172.21.15.135:3300/0 conn(0x7fd5d00a5fb0 0x7fd5d0141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:51.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.549+0000 7fd5cffff700 1 --2- 172.21.15.135:0/1365900838 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5d00b1460 0x7fd5d00b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:51.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.549+0000 7fd5cf7fe700 10 monclient(hunting): get_auth_request con 0x7fd5d00a5fb0 auth_method 0 2023-09-30T00:46:51.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.549+0000 7fd5cf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:51.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.549+0000 7fd5cf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:51.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.549+0000 7fd5cf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:51.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.550+0000 7fd5cffff700 10 monclient(hunting): get_auth_request con 0x7fd5d00b1460 auth_method 0 2023-09-30T00:46:51.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.550+0000 7fd5cffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:51.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.550+0000 7fd5cffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:51.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.550+0000 7fd5cffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:51.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.550+0000 7fd5cf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:51.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.550+0000 7fd5cf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:51.555 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.550+0000 7fd5cf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:51.555 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.550+0000 7fd5cffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:51.555 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.550+0000 7fd5cffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:51.555 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.550+0000 7fd5cffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:51.555 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.552+0000 7fd5cf7fe700 10 monclient(hunting): handle_auth_done global_id 4144 payload 274 2023-09-30T00:46:51.556 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.552+0000 7fd5cf7fe700 1 -- 172.21.15.135:0/1365900838 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5d00b1460 msgr2=0x7fd5d00b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:51.556 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.552+0000 7fd5cf7fe700 1 --2- 172.21.15.135:0/1365900838 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5d00b1460 0x7fd5d00b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.556 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.552+0000 7fd5cf7fe700 1 -- 172.21.15.135:0/1365900838 >> v1:172.21.15.40:6789/0 conn(0x7fd5d00af410 legacy=0x7fd5d00af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:51.556 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.552+0000 7fd5cf7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:46:51.556 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.552+0000 7fd5cf7fe700 1 monclient: found mon.noname-c 2023-09-30T00:46:51.556 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.552+0000 7fd5cf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:51.556 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.552+0000 7fd5cf7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:46:51.556 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.552+0000 7fd5cf7fe700 1 -- 172.21.15.135:0/1365900838 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d00a56d0 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.556 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.554+0000 7fd5cf7fe700 1 --2- 172.21.15.135:0/1365900838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d00a5fb0 0x7fd5d0141730 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd5c40098e0 tx=0x7fd5c40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df313186309399cf server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:51.559 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.557+0000 7fd5ceffd700 1 -- 172.21.15.135:0/1365900838 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c400b640 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.559 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.557+0000 7fd5ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:51.559 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.557+0000 7fd5ceffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:51.559 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.557+0000 7fd5ceffd700 10 monclient: dump: 2023-09-30T00:46:51.559 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:51.559 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:51.559 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.560 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.560 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:51.560 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:51.560 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:51.560 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:51.560 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:51.560 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:51.560 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.557+0000 7fd5ceffd700 10 monclient: _finish_auth 0 2023-09-30T00:46:51.560 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.557+0000 7fd5ceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:51.560 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.557+0000 7fd5d6944700 5 monclient: authenticate success, global_id 4144 2023-09-30T00:46:51.560 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.557+0000 7fd5ceffd700 1 -- 172.21.15.135:0/1365900838 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c400b7a0 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.561 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.557+0000 7fd5d6944700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:51.561 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.557+0000 7fd5ceffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:51.561 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.557+0000 7fd5d6944700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:51.561 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.557+0000 7fd5d6944700 4 set_mon_vals no callback set 2023-09-30T00:46:51.561 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.557+0000 7fd5ceffd700 1 -- 172.21.15.135:0/1365900838 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c400bb40 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.561 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.558+0000 7fd5ceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:51.561 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.558+0000 7fd5ceffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:51.561 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.558+0000 7fd5ceffd700 10 monclient: dump: 2023-09-30T00:46:51.561 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:51.561 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:51.561 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.561 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.562 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:51.562 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:51.562 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:51.562 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:51.562 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:51.562 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:51.565 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.558+0000 7fd5d6944700 10 monclient: shutdown 2023-09-30T00:46:51.565 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.558+0000 7fd5d6944700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:51.566 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.558+0000 7fd5d6944700 1 -- 172.21.15.135:0/1365900838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d00a5fb0 msgr2=0x7fd5d0141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:51.566 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.558+0000 7fd5d6944700 1 --2- 172.21.15.135:0/1365900838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d00a5fb0 0x7fd5d0141730 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd5c40098e0 tx=0x7fd5c40091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.566 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.558+0000 7fd5d6944700 1 -- 172.21.15.135:0/1365900838 shutdown_connections 2023-09-30T00:46:51.566 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.558+0000 7fd5d6944700 1 --2- 172.21.15.135:0/1365900838 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5d00b1460 0x7fd5d00b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.566 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.558+0000 7fd5d6944700 1 --2- 172.21.15.135:0/1365900838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d00a5fb0 0x7fd5d0141730 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.566 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.558+0000 7fd5d6944700 1 -- 172.21.15.135:0/1365900838 >> 172.21.15.135:0/1365900838 conn(0x7fd5d00631f0 msgr2=0x7fd5d00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:51.566 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.558+0000 7fd5d6944700 1 -- 172.21.15.135:0/1365900838 shutdown_connections 2023-09-30T00:46:51.566 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.558+0000 7fd5d6944700 1 -- 172.21.15.135:0/1365900838 wait complete. 2023-09-30T00:46:51.566 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.559+0000 7fd5d6944700 10 monclient: build_initial_monmap 2023-09-30T00:46:51.566 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.559+0000 7fd5d6944700 10 monclient: monmap: 2023-09-30T00:46:51.566 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:51.567 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:51.567 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2023-09-30T00:46:51.567 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2023-09-30T00:46:51.567 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:51.567 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2023-09-30T00:46:51.567 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:46:51.567 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:46:51.567 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:46:51.567 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:51.567 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.559+0000 7fd5d6944700 1 librados: starting msgr at 2023-09-30T00:46:51.567 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.559+0000 7fd5d6944700 1 librados: starting objecter 2023-09-30T00:46:51.567 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 1 Processor -- start 2023-09-30T00:46:51.567 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 1 -- start start 2023-09-30T00:46:51.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 1 librados: setting wanted keys 2023-09-30T00:46:51.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 1 librados: calling monclient init 2023-09-30T00:46:51.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 10 monclient: init 2023-09-30T00:46:51.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:51.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:46:51.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d00a5fb0 0x7fd5d014e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:51.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 10 monclient(hunting): picked mon.noname-b con 0x7fd5d00a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:46:51.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5d00af410 0x7fd5d00a7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:51.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 10 monclient(hunting): picked mon.noname-c con 0x7fd5d00af410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:51.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d00b1460 0x7fd5d00a8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:51.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 10 monclient(hunting): picked mon.noname-a con 0x7fd5d00b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:46:51.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:51.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00a8170 con 0x7fd5d00b1460 2023-09-30T00:46:51.569 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:51.569 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00a9d30 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.569 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:51.569 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d014eec0 con 0x7fd5d00af410 2023-09-30T00:46:51.569 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:51.569 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.560+0000 7fd5d6944700 10 monclient(hunting): authenticate will time out at 5279.999852s 2023-09-30T00:46:51.569 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5cffff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d00a5fb0 0x7fd5d014e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:51.569 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5d4ee1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d00b1460 0x7fd5d00a8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:51.569 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5cf7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5d00af410 0x7fd5d00a7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:51.569 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5cffff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d00a5fb0 0x7fd5d014e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:54212/0 (socket says 172.21.15.135:54212) 2023-09-30T00:46:51.569 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5cffff700 1 -- 172.21.15.135:0/2587677798 learned_addr learned my addr 172.21.15.135:0/2587677798 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-09-30T00:46:51.569 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5cffff700 10 monclient(hunting): get_auth_request con 0x7fd5d00a5fb0 auth_method 0 2023-09-30T00:46:51.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5cffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:51.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5cffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:51.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5cffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:51.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5d4ee1700 10 monclient(hunting): get_auth_request con 0x7fd5d00b1460 auth_method 0 2023-09-30T00:46:51.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5d4ee1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:51.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5d4ee1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:51.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5d4ee1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:51.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5cf7fe700 10 monclient(hunting): get_auth_request con 0x7fd5d00af410 auth_method 0 2023-09-30T00:46:51.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5cf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:51.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5cf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:51.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.561+0000 7fd5cf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:51.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.562+0000 7fd5cffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:51.571 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.562+0000 7fd5cffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:51.571 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.562+0000 7fd5cffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:51.571 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.562+0000 7fd5d4ee1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:51.571 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.562+0000 7fd5d4ee1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:51.571 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.562+0000 7fd5d4ee1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:51.571 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.562+0000 7fd5cf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:51.571 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.562+0000 7fd5cf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:51.571 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.562+0000 7fd5cf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:51.573 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.565+0000 7fd5cffff700 10 monclient(hunting): handle_auth_done global_id 4147 payload 824 2023-09-30T00:46:51.574 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.565+0000 7fd5cffff700 1 -- 172.21.15.135:0/2587677798 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5d00af410 msgr2=0x7fd5d00a7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:51.574 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.565+0000 7fd5cffff700 1 --2- 172.21.15.135:0/2587677798 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5d00af410 0x7fd5d00a7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.574 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.565+0000 7fd5cffff700 1 -- 172.21.15.135:0/2587677798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d00b1460 msgr2=0x7fd5d00a8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:51.574 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.565+0000 7fd5cffff700 1 --2- 172.21.15.135:0/2587677798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d00b1460 0x7fd5d00a8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.574 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.565+0000 7fd5cffff700 10 monclient: _finish_hunting 0 2023-09-30T00:46:51.574 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.565+0000 7fd5cffff700 1 monclient: found mon.noname-b 2023-09-30T00:46:51.574 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.565+0000 7fd5cffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:51.574 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.565+0000 7fd5cffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:46:51.574 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.565+0000 7fd5cffff700 1 -- 172.21.15.135:0/2587677798 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c4009420 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.574 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.567+0000 7fd5cffff700 1 --2- 172.21.15.135:0/2587677798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d00a5fb0 0x7fd5d014e450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000e960 tx=0x7fd5c00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:51.574 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.567+0000 7fd5b7fff700 10 client.?.objecter ms_handle_connect 0x7fd5d00a5fb0 2023-09-30T00:46:51.575 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.567+0000 7fd5b7fff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:46:51.575 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5b7fff700 1 -- 172.21.15.135:0/2587677798 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0013680 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.575 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:51.575 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5b7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:51.575 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5b7fff700 10 monclient: dump: 2023-09-30T00:46:51.575 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:51.575 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:51.575 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.575 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.575 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:51.575 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:51.575 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:51.575 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:51.575 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:51.576 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:51.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5b7fff700 10 monclient: _finish_auth 0 2023-09-30T00:46:51.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5b7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:51.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5b7fff700 1 -- 172.21.15.135:0/2587677798 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c00137e0 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5d6944700 5 monclient: authenticate success, global_id 4147 2023-09-30T00:46:51.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5d6944700 10 monclient: _renew_subs 2023-09-30T00:46:51.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5d6944700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:51.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5d6944700 1 -- 172.21.15.135:0/2587677798 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d014f030 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5b7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:51.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5b7fff700 1 -- 172.21.15.135:0/2587677798 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0013ba0 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:51.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5b7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:51.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5b7fff700 10 monclient: dump: 2023-09-30T00:46:51.577 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:51.577 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:51.577 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.577 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:51.577 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:51.577 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:51.577 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:51.577 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:51.577 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:51.577 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:51.577 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5ce7fc700 4 set_mon_vals no callback set 2023-09-30T00:46:51.577 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5d6944700 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:46:51.577 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5d6944700 10 monclient: _renew_subs 2023-09-30T00:46:51.578 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5d6944700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:51.578 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5d6944700 1 -- 172.21.15.135:0/2587677798 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d00a8b40 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.578 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.571+0000 7fd5d6944700 1 librados: init done 2023-09-30T00:46:51.578 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.572+0000 7fd5d6944700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:51.578 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.572+0000 7fd5d6944700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:51.578 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.572+0000 7fd5d6944700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:51.578 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.572+0000 7fd5d6944700 1 -- 172.21.15.135:0/2587677798 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d00a7f50 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.579 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.573+0000 7fd5b7fff700 1 -- 172.21.15.135:0/2587677798 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd5c0005c80 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.579 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.574+0000 7fd5b7fff700 1 -- 172.21.15.135:0/2587677798 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7fd5c0005f30 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.579 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.574+0000 7fd5b7fff700 10 client.4147.objecter ms_dispatch 0x7fd5d0001550 osd_map(7..7 src has 1..7) v4 2023-09-30T00:46:51.579 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.574+0000 7fd5b7fff700 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2023-09-30T00:46:51.579 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.574+0000 7fd5b7fff700 3 client.4147.objecter handle_osd_map decoding full epoch 7 2023-09-30T00:46:51.579 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.574+0000 7fd5b7fff700 20 client.4147.objecter dump_active .. 0 homeless 2023-09-30T00:46:51.589 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:51.588+0000 7fadf12501c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T00:46:51.665 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:51.664+0000 7feafc14a1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T00:46:51.678 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.677+0000 7fd5b7fff700 1 -- 172.21.15.135:0/2587677798 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fd5c0021060 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.678 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.677+0000 7fd5b7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:51.678 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.677+0000 7fd5b7fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:51.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.760+0000 7fd5d6944700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7ea449db-9b22-48b8-a61b-1937eeeb6c8e", "id": 5}] 2023-09-30T00:46:51.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.760+0000 7fd5d6944700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7ea449db-9b22-48b8-a61b-1937eeeb6c8e", "id": 5}] 2023-09-30T00:46:51.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.760+0000 7fd5d6944700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:51.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.760+0000 7fd5d6944700 1 -- 172.21.15.135:0/2587677798 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7ea449db-9b22-48b8-a61b-1937eeeb6c8e", "id": 5} v 0) v1 -- 0x7fd5d00a8450 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.855 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.853+0000 7fd5b7fff700 1 -- 172.21.15.135:0/2587677798 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7ea449db-9b22-48b8-a61b-1937eeeb6c8e", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd5c0017070 con 0x7fd5d00a5fb0 2023-09-30T00:46:51.855 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.853+0000 7fd5b7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7ea449db-9b22-48b8-a61b-1937eeeb6c8e", "id": 5}] 2023-09-30T00:46:51.855 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.853+0000 7fd5b7fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:51.855 INFO:teuthology.orchestra.run.smithi135.stdout:5 2023-09-30T00:46:51.856 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.854+0000 7fd5d6944700 10 librados: watch_flush enter 2023-09-30T00:46:51.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.854+0000 7fd5d6944700 10 librados: watch_flush exit 2023-09-30T00:46:51.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.854+0000 7fd5d6944700 20 client.4147.objecter shutdown clearing up homeless session... 2023-09-30T00:46:51.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.854+0000 7fd5d6944700 10 client.4147.objecter successfully canceled tick 2023-09-30T00:46:51.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.855+0000 7fd5d6944700 10 monclient: shutdown 2023-09-30T00:46:51.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.855+0000 7fd5d6944700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:51.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.855+0000 7fd5d6944700 1 -- 172.21.15.135:0/2587677798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d00a5fb0 msgr2=0x7fd5d014e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:51.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.855+0000 7fd5d6944700 1 --2- 172.21.15.135:0/2587677798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d00a5fb0 0x7fd5d014e450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000e960 tx=0x7fd5c00091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.855+0000 7fd5d6944700 1 -- 172.21.15.135:0/2587677798 shutdown_connections 2023-09-30T00:46:51.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.855+0000 7fd5d6944700 1 --2- 172.21.15.135:0/2587677798 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5d00a5fb0 0x7fd5d014e450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.855+0000 7fd5d6944700 1 --2- 172.21.15.135:0/2587677798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5d00b1460 0x7fd5d00a8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.855+0000 7fd5d6944700 1 --2- 172.21.15.135:0/2587677798 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5d00af410 0x7fd5d00a7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.855+0000 7fd5d6944700 1 -- 172.21.15.135:0/2587677798 >> 172.21.15.135:0/2587677798 conn(0x7fd5d00631f0 msgr2=0x7fd5d01399c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:51.858 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.855+0000 7fd5d6944700 1 -- 172.21.15.135:0/2587677798 shutdown_connections 2023-09-30T00:46:51.858 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.855+0000 7fd5d6944700 1 -- 172.21.15.135:0/2587677798 wait complete. 2023-09-30T00:46:51.858 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.855+0000 7fd5d6944700 1 librados: shutdown 2023-09-30T00:46:51.868 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new c79b4829-6393-4f9a-a9f8-d2aacb505ae9 6 2023-09-30T00:46:51.926 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:51.925+0000 7fadf12501c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T00:46:51.990 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.989+0000 7fddd2f4d700 10 monclient: get_monmap_and_config 2023-09-30T00:46:51.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.989+0000 7fddd2f4d700 10 monclient: build_initial_monmap 2023-09-30T00:46:51.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.989+0000 7fddd2f4d700 10 monclient: monmap: 2023-09-30T00:46:51.991 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:51.991 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:51.991 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:46:51.991396+0000 2023-09-30T00:46:51.991 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:46:51.991396+0000 2023-09-30T00:46:51.991 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:51.991 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:51.991 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:46:51.991 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:46:51.991 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:46:51.992 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:46:51.992 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:46:51.992 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:51.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.990+0000 7fddd2f4d700 1 Processor -- start 2023-09-30T00:46:51.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.990+0000 7fddd2f4d700 1 -- start start 2023-09-30T00:46:51.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.990+0000 7fddd2f4d700 10 monclient: init 2023-09-30T00:46:51.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddd2f4d700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:51.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddd2f4d700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-09-30T00:46:51.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddd2f4d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fddcc0b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:46:51.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddd2f4d700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fddcc0b1830 0x7fddcc0a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:51.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddd2f4d700 10 monclient(hunting): picked mon.noname-c con 0x7fddcc0b1830 addr v2:172.21.15.135:3300/0 2023-09-30T00:46:51.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddd2f4d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fddcc13ac20 0x7fddcc13b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:51.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddd2f4d700 10 monclient(hunting): picked mon.noname-a con 0x7fddcc13ac20 addr v2:172.21.15.40:3300/0 2023-09-30T00:46:51.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddd2f4d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddcc0a5b20 con 0x7fddcc0b1460 2023-09-30T00:46:51.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddd2f4d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:51.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddd2f4d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddcc141e40 con 0x7fddcc13ac20 2023-09-30T00:46:51.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddd2f4d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:51.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddd2f4d700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddcc0a64f0 con 0x7fddcc0b1830 2023-09-30T00:46:51.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddd2f4d700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:51.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddd2f4d700 10 monclient(hunting): authenticate will time out at 5280.430183s 2023-09-30T00:46:51.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.991+0000 7fddcbfff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fddcc0b1830 0x7fddcc0a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:51.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.992+0000 7fddcbfff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fddcc0b1830 0x7fddcc0a5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:54222/0 (socket says 172.21.15.135:54222) 2023-09-30T00:46:51.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.992+0000 7fddcbfff700 1 -- 172.21.15.135:0/4275966547 learned_addr learned my addr 172.21.15.135:0/4275966547 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-09-30T00:46:51.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.992+0000 7fddcbfff700 10 monclient(hunting): get_auth_request con 0x7fddcc0b1830 auth_method 0 2023-09-30T00:46:51.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.992+0000 7fddcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:51.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.992+0000 7fddcbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:51.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.992+0000 7fddcbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:51.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.992+0000 7fddd14ea700 1 --2- 172.21.15.135:0/4275966547 >> v2:172.21.15.40:3300/0 conn(0x7fddcc13ac20 0x7fddcc13b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:51.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.992+0000 7fddd14ea700 10 monclient(hunting): get_auth_request con 0x7fddcc13ac20 auth_method 0 2023-09-30T00:46:51.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.992+0000 7fddd14ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:51.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.992+0000 7fddd14ea700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:51.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.992+0000 7fddd14ea700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:51.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.992+0000 7fddcbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:51.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.992+0000 7fddcbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:51.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.992+0000 7fddcbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:51.997 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.993+0000 7fddd14ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:51.997 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.993+0000 7fddd14ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:51.997 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.993+0000 7fddd14ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:51.997 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.994+0000 7fddcbfff700 10 monclient(hunting): handle_auth_done global_id 4150 payload 274 2023-09-30T00:46:51.997 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.994+0000 7fddcbfff700 1 -- 172.21.15.135:0/4275966547 >> v2:172.21.15.40:3300/0 conn(0x7fddcc13ac20 msgr2=0x7fddcc13b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:51.997 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.994+0000 7fddcbfff700 1 --2- 172.21.15.135:0/4275966547 >> v2:172.21.15.40:3300/0 conn(0x7fddcc13ac20 0x7fddcc13b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:51.997 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.994+0000 7fddcbfff700 1 -- 172.21.15.135:0/4275966547 >> v1:172.21.15.40:6789/0 conn(0x7fddcc0b1460 legacy=0x7fddcc141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:51.998 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.994+0000 7fddcbfff700 10 monclient: _finish_hunting 0 2023-09-30T00:46:51.998 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.994+0000 7fddcbfff700 1 monclient: found mon.noname-c 2023-09-30T00:46:51.998 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.994+0000 7fddcbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:51.998 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.994+0000 7fddcbfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:46:51.998 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.994+0000 7fddcbfff700 1 -- 172.21.15.135:0/4275966547 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddcc0a71f0 con 0x7fddcc0b1830 2023-09-30T00:46:51.998 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:51.996+0000 7fddcbfff700 1 --2- 172.21.15.135:0/4275966547 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc0b1830 0x7fddcc0a5fb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fddb80099b0 tx=0x7fddb8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dfbcd6a04e6ff0c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:52.002 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddcb7fe700 1 -- 172.21.15.135:0/4275966547 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8009ea0 con 0x7fddcc0b1830 2023-09-30T00:46:52.002 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddcb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:52.002 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddcb7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:52.002 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddcb7fe700 10 monclient: dump: 2023-09-30T00:46:52.003 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:52.003 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:52.003 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.003 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.003 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:52.003 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:52.003 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:52.003 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:52.003 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:52.003 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:52.003 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddcb7fe700 10 monclient: _finish_auth 0 2023-09-30T00:46:52.003 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddcb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:52.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddcb7fe700 1 -- 172.21.15.135:0/4275966547 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb800b820 con 0x7fddcc0b1830 2023-09-30T00:46:52.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddd2f4d700 5 monclient: authenticate success, global_id 4150 2023-09-30T00:46:52.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddd2f4d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:52.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddcb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:52.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddd2f4d700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:52.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddd2f4d700 4 set_mon_vals no callback set 2023-09-30T00:46:52.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddcb7fe700 1 -- 172.21.15.135:0/4275966547 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb800bbc0 con 0x7fddcc0b1830 2023-09-30T00:46:52.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddcb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:52.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddcb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:52.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.000+0000 7fddcb7fe700 10 monclient: dump: 2023-09-30T00:46:52.004 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:52.004 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:52.005 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.005 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.005 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:52.005 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:52.005 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:52.005 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:52.005 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:52.005 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:52.005 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.001+0000 7fddd2f4d700 10 monclient: shutdown 2023-09-30T00:46:52.005 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.001+0000 7fddd2f4d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:52.005 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.001+0000 7fddd2f4d700 1 -- 172.21.15.135:0/4275966547 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc0b1830 msgr2=0x7fddcc0a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:52.005 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.001+0000 7fddd2f4d700 1 --2- 172.21.15.135:0/4275966547 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc0b1830 0x7fddcc0a5fb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fddb80099b0 tx=0x7fddb8009250 comp rx=0 tx=0).stop 2023-09-30T00:46:52.006 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.001+0000 7fddd2f4d700 1 -- 172.21.15.135:0/4275966547 shutdown_connections 2023-09-30T00:46:52.006 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.001+0000 7fddd2f4d700 1 --2- 172.21.15.135:0/4275966547 >> v2:172.21.15.40:3300/0 conn(0x7fddcc13ac20 0x7fddcc13b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.006 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.001+0000 7fddd2f4d700 1 --2- 172.21.15.135:0/4275966547 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc0b1830 0x7fddcc0a5fb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.006 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.001+0000 7fddd2f4d700 1 -- 172.21.15.135:0/4275966547 >> 172.21.15.135:0/4275966547 conn(0x7fddcc0631f0 msgr2=0x7fddcc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:52.006 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:52.001+0000 7feafc14a1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T00:46:52.008 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:52.003+0000 7fadf12501c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T00:46:52.009 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.001+0000 7fddd2f4d700 1 -- 172.21.15.135:0/4275966547 shutdown_connections 2023-09-30T00:46:52.009 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.001+0000 7fddd2f4d700 1 -- 172.21.15.135:0/4275966547 wait complete. 2023-09-30T00:46:52.010 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.002+0000 7fddd2f4d700 10 monclient: build_initial_monmap 2023-09-30T00:46:52.010 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.002+0000 7fddd2f4d700 10 monclient: monmap: 2023-09-30T00:46:52.010 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:52.010 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:52.010 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2023-09-30T00:46:52.010 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2023-09-30T00:46:52.010 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:52.010 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2023-09-30T00:46:52.010 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:46:52.010 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:46:52.010 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:46:52.011 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:52.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.003+0000 7fddd2f4d700 1 librados: starting msgr at 2023-09-30T00:46:52.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.003+0000 7fddd2f4d700 1 librados: starting objecter 2023-09-30T00:46:52.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.003+0000 7fddd2f4d700 1 Processor -- start 2023-09-30T00:46:52.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.003+0000 7fddd2f4d700 1 -- start start 2023-09-30T00:46:52.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.003+0000 7fddd2f4d700 1 librados: setting wanted keys 2023-09-30T00:46:52.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.003+0000 7fddd2f4d700 1 librados: calling monclient init 2023-09-30T00:46:52.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.003+0000 7fddd2f4d700 10 monclient: init 2023-09-30T00:46:52.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:52.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:46:52.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddcc0b1460 0x7fddcc13ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 10 monclient(hunting): picked mon.noname-a con 0x7fddcc0b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:46:52.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc0b1830 0x7fddcc151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.012 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 10 monclient(hunting): picked mon.noname-b con 0x7fddcc0b1830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:46:52.012 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddcc13ac20 0x7fddcc0a79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.012 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 10 monclient(hunting): picked mon.noname-c con 0x7fddcc13ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:52.012 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:52.012 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc0a9b20 con 0x7fddcc0b1460 2023-09-30T00:46:52.012 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:52.012 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc0aa820 con 0x7fddcc0b1830 2023-09-30T00:46:52.012 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:52.012 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc0aed20 con 0x7fddcc13ac20 2023-09-30T00:46:52.012 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:52.012 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.004+0000 7fddd2f4d700 10 monclient(hunting): authenticate will time out at 5280.443675s 2023-09-30T00:46:52.013 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddd0ce9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddcc0b1460 0x7fddcc13ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.013 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddcbfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc0b1830 0x7fddcc151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.013 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddd14ea700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddcc13ac20 0x7fddcc0a79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.013 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddd0ce9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddcc0b1460 0x7fddcc13ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.135:52740/0 (socket says 172.21.15.135:52740) 2023-09-30T00:46:52.013 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddd0ce9700 1 -- 172.21.15.135:0/899145826 learned_addr learned my addr 172.21.15.135:0/899145826 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-09-30T00:46:52.013 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddd0ce9700 10 monclient(hunting): get_auth_request con 0x7fddcc0b1460 auth_method 0 2023-09-30T00:46:52.013 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddd0ce9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:52.013 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddd0ce9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:52.013 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddd0ce9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:52.013 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddcbfff700 10 monclient(hunting): get_auth_request con 0x7fddcc0b1830 auth_method 0 2023-09-30T00:46:52.013 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddcbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:52.013 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddcbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:52.013 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddcbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:52.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddd14ea700 10 monclient(hunting): get_auth_request con 0x7fddcc13ac20 auth_method 0 2023-09-30T00:46:52.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddd14ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:52.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddd14ea700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:52.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.005+0000 7fddd14ea700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:52.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.006+0000 7fddd0ce9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:52.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.006+0000 7fddd0ce9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:52.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.006+0000 7fddd0ce9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:52.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.006+0000 7fddcbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:52.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.006+0000 7fddcbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:52.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.006+0000 7fddcbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:52.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.006+0000 7fddd14ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:52.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.006+0000 7fddd14ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:52.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.006+0000 7fddd14ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:52.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.009+0000 7fddcbfff700 10 monclient(hunting): handle_auth_done global_id 4153 payload 824 2023-09-30T00:46:52.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.009+0000 7fddcbfff700 1 -- 172.21.15.135:0/899145826 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddcc13ac20 msgr2=0x7fddcc0a79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:52.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.009+0000 7fddcbfff700 1 --2- 172.21.15.135:0/899145826 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddcc13ac20 0x7fddcc0a79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.009+0000 7fddcbfff700 1 -- 172.21.15.135:0/899145826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddcc0b1460 msgr2=0x7fddcc13ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:52.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.009+0000 7fddcbfff700 1 --2- 172.21.15.135:0/899145826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddcc0b1460 0x7fddcc13ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.009+0000 7fddcbfff700 10 monclient: _finish_hunting 0 2023-09-30T00:46:52.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.009+0000 7fddcbfff700 1 monclient: found mon.noname-b 2023-09-30T00:46:52.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.009+0000 7fddcbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:52.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.009+0000 7fddcbfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:46:52.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.009+0000 7fddcbfff700 1 -- 172.21.15.135:0/899145826 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb80094f0 con 0x7fddcc0b1830 2023-09-30T00:46:52.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.011+0000 7fddcbfff700 1 --2- 172.21.15.135:0/899145826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc0b1830 0x7fddcc151230 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fddb8016000 tx=0x7fddb8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:52.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.011+0000 7fddc8ff9700 10 client.?.objecter ms_handle_connect 0x7fddcc0b1830 2023-09-30T00:46:52.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.011+0000 7fddc8ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:46:52.018 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddc8ff9700 1 -- 172.21.15.135:0/899145826 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8000d10 con 0x7fddcc0b1830 2023-09-30T00:46:52.019 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddc8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:52.019 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddc8ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:52.019 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddc8ff9700 10 monclient: dump: 2023-09-30T00:46:52.019 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:52.019 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:52.019 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.019 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.019 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:52.019 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:52.019 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:52.019 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:52.019 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:52.020 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:52.020 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddc8ff9700 10 monclient: _finish_auth 0 2023-09-30T00:46:52.020 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddc8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:52.020 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddc8ff9700 1 -- 172.21.15.135:0/899145826 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddb800bba0 con 0x7fddcc0b1830 2023-09-30T00:46:52.020 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddc8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:52.020 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddc8ff9700 1 -- 172.21.15.135:0/899145826 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb8011670 con 0x7fddcc0b1830 2023-09-30T00:46:52.020 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddd2f4d700 5 monclient: authenticate success, global_id 4153 2023-09-30T00:46:52.020 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddd2f4d700 10 monclient: _renew_subs 2023-09-30T00:46:52.020 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddcaffd700 4 set_mon_vals no callback set 2023-09-30T00:46:52.020 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddd2f4d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:52.020 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddd2f4d700 1 -- 172.21.15.135:0/899145826 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddcc0aa280 con 0x7fddcc0b1830 2023-09-30T00:46:52.020 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddc8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:52.021 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddc8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:52.021 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddc8ff9700 10 monclient: dump: 2023-09-30T00:46:52.021 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:52.021 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:52.021 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.021 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.021 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:52.021 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:52.021 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:52.021 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:52.021 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:52.021 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:52.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddd2f4d700 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:46:52.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddd2f4d700 10 monclient: _renew_subs 2023-09-30T00:46:52.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddd2f4d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:52.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddd2f4d700 1 -- 172.21.15.135:0/899145826 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddcc151770 con 0x7fddcc0b1830 2023-09-30T00:46:52.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.015+0000 7fddd2f4d700 1 librados: init done 2023-09-30T00:46:52.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.016+0000 7fddd2f4d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:52.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.016+0000 7fddd2f4d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:52.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.016+0000 7fddd2f4d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:52.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.016+0000 7fddd2f4d700 1 -- 172.21.15.135:0/899145826 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddcc0aaa60 con 0x7fddcc0b1830 2023-09-30T00:46:52.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.017+0000 7fddc8ff9700 1 -- 172.21.15.135:0/899145826 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fddb80119e0 con 0x7fddcc0b1830 2023-09-30T00:46:52.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.018+0000 7fddc8ff9700 1 -- 172.21.15.135:0/899145826 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7fddb8011c90 con 0x7fddcc0b1830 2023-09-30T00:46:52.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.018+0000 7fddc8ff9700 10 client.4153.objecter ms_dispatch 0x7fddcc001550 osd_map(8..8 src has 1..8) v4 2023-09-30T00:46:52.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.018+0000 7fddc8ff9700 3 client.4153.objecter handle_osd_map got epochs [8,8] > 0 2023-09-30T00:46:52.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.018+0000 7fddc8ff9700 3 client.4153.objecter handle_osd_map decoding full epoch 8 2023-09-30T00:46:52.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.019+0000 7fddc8ff9700 20 client.4153.objecter dump_active .. 0 homeless 2023-09-30T00:46:52.123 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.121+0000 7fddc8ff9700 1 -- 172.21.15.135:0/899145826 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fddcc0aaa60 con 0x7fddcc0b1830 2023-09-30T00:46:52.123 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.121+0000 7fddc8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:52.123 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.121+0000 7fddc8ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:52.148 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:52.147+0000 7feafc14a1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:46:52.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.201+0000 7fddd2f4d700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c79b4829-6393-4f9a-a9f8-d2aacb505ae9", "id": 6}] 2023-09-30T00:46:52.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.201+0000 7fddd2f4d700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c79b4829-6393-4f9a-a9f8-d2aacb505ae9", "id": 6}] 2023-09-30T00:46:52.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.201+0000 7fddd2f4d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:52.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.201+0000 7fddd2f4d700 1 -- 172.21.15.135:0/899145826 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c79b4829-6393-4f9a-a9f8-d2aacb505ae9", "id": 6} v 0) v1 -- 0x7fddcc064a00 con 0x7fddcc0b1830 2023-09-30T00:46:52.300 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.298+0000 7fddc8ff9700 1 -- 172.21.15.135:0/899145826 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c79b4829-6393-4f9a-a9f8-d2aacb505ae9", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fddb8028070 con 0x7fddcc0b1830 2023-09-30T00:46:52.300 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.298+0000 7fddc8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c79b4829-6393-4f9a-a9f8-d2aacb505ae9", "id": 6}] 2023-09-30T00:46:52.300 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.298+0000 7fddc8ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:52.300 INFO:teuthology.orchestra.run.smithi135.stdout:6 2023-09-30T00:46:52.301 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.299+0000 7fddd2f4d700 10 librados: watch_flush enter 2023-09-30T00:46:52.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.299+0000 7fddd2f4d700 10 librados: watch_flush exit 2023-09-30T00:46:52.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.299+0000 7fddd2f4d700 20 client.4153.objecter shutdown clearing up homeless session... 2023-09-30T00:46:52.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.299+0000 7fddd2f4d700 10 client.4153.objecter successfully canceled tick 2023-09-30T00:46:52.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.299+0000 7fddd2f4d700 10 monclient: shutdown 2023-09-30T00:46:52.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.299+0000 7fddd2f4d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:52.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.299+0000 7fddd2f4d700 1 -- 172.21.15.135:0/899145826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc0b1830 msgr2=0x7fddcc151230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:52.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.299+0000 7fddd2f4d700 1 --2- 172.21.15.135:0/899145826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc0b1830 0x7fddcc151230 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fddb8016000 tx=0x7fddb8017000 comp rx=0 tx=0).stop 2023-09-30T00:46:52.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.299+0000 7fddd2f4d700 1 -- 172.21.15.135:0/899145826 shutdown_connections 2023-09-30T00:46:52.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.299+0000 7fddd2f4d700 1 --2- 172.21.15.135:0/899145826 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fddcc13ac20 0x7fddcc0a79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.299+0000 7fddd2f4d700 1 --2- 172.21.15.135:0/899145826 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddcc0b1460 0x7fddcc13ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.299+0000 7fddd2f4d700 1 --2- 172.21.15.135:0/899145826 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fddcc0b1830 0x7fddcc151230 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.303 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.299+0000 7fddd2f4d700 1 -- 172.21.15.135:0/899145826 >> 172.21.15.135:0/899145826 conn(0x7fddcc0631f0 msgr2=0x7fddcc139d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:52.303 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.300+0000 7fddd2f4d700 1 -- 172.21.15.135:0/899145826 shutdown_connections 2023-09-30T00:46:52.303 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.300+0000 7fddd2f4d700 1 -- 172.21.15.135:0/899145826 wait complete. 2023-09-30T00:46:52.303 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.300+0000 7fddd2f4d700 1 librados: shutdown 2023-09-30T00:46:52.313 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd new 4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7 7 2023-09-30T00:46:52.430 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.429+0000 7fb818a08700 10 monclient: get_monmap_and_config 2023-09-30T00:46:52.431 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.429+0000 7fb818a08700 10 monclient: build_initial_monmap 2023-09-30T00:46:52.431 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.429+0000 7fb818a08700 10 monclient: monmap: 2023-09-30T00:46:52.431 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:52.431 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:52.431 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:46:52.430815+0000 2023-09-30T00:46:52.431 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:46:52.430815+0000 2023-09-30T00:46:52.431 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:52.431 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:52.431 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:46:52.431 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:46:52.431 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:46:52.432 INFO:teuthology.orchestra.run.smithi135.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:46:52.432 INFO:teuthology.orchestra.run.smithi135.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:46:52.432 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:52.432 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.429+0000 7fb818a08700 1 Processor -- start 2023-09-30T00:46:52.432 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.429+0000 7fb818a08700 1 -- start start 2023-09-30T00:46:52.432 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.429+0000 7fb818a08700 10 monclient: init 2023-09-30T00:46:52.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:52.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-09-30T00:46:52.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb8140a5710 0x7fb8140a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 10 monclient(hunting): picked mon.noname-c con 0x7fb8140a5710 addr v2:172.21.15.135:3300/0 2023-09-30T00:46:52.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8140a7b20 0x7fb81413e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 10 monclient(hunting): picked mon.noname-b con 0x7fb8140a7b20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:52.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb81413e710 0x7fb814141760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 10 monclient(hunting): picked mon.noname-a con 0x7fb81413e710 addr v2:172.21.15.40:3300/0 2023-09-30T00:46:52.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:52.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8140a5ae0 con 0x7fb81413e710 2023-09-30T00:46:52.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:52.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8140a7ff0 con 0x7fb8140a5710 2023-09-30T00:46:52.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:52.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8140aedc0 con 0x7fb8140a7b20 2023-09-30T00:46:52.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:52.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.430+0000 7fb818a08700 10 monclient(hunting): authenticate will time out at 5280.869619s 2023-09-30T00:46:52.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb81259c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb8140a5710 0x7fb8140a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb812d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb81413e710 0x7fb814141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb811d9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8140a7b20 0x7fb81413e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb812d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb81413e710 0x7fb814141760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.135:52746/0 (socket says 172.21.15.135:52746) 2023-09-30T00:46:52.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb812d9d700 1 -- 172.21.15.135:0/1377433696 learned_addr learned my addr 172.21.15.135:0/1377433696 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-09-30T00:46:52.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb812d9d700 10 monclient(hunting): get_auth_request con 0x7fb81413e710 auth_method 0 2023-09-30T00:46:52.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb812d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:52.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb812d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:52.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb812d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:52.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb81259c700 10 monclient(hunting): get_auth_request con 0x7fb8140a5710 auth_method 0 2023-09-30T00:46:52.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb81259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:52.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb81259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:52.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb81259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:52.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb811d9b700 10 monclient(hunting): get_auth_request con 0x7fb8140a7b20 auth_method 0 2023-09-30T00:46:52.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb811d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:52.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb811d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:52.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.431+0000 7fb811d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:52.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.432+0000 7fb81259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:52.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.432+0000 7fb81259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:52.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.432+0000 7fb81259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:52.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.432+0000 7fb812d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:52.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.432+0000 7fb812d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:52.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.432+0000 7fb812d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:52.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.432+0000 7fb811d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:52.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.432+0000 7fb811d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:52.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.432+0000 7fb811d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:52.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.434+0000 7fb81259c700 10 monclient(hunting): handle_auth_done global_id 4156 payload 274 2023-09-30T00:46:52.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.434+0000 7fb81259c700 1 -- 172.21.15.135:0/1377433696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8140a7b20 msgr2=0x7fb81413e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:52.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.434+0000 7fb81259c700 1 --2- 172.21.15.135:0/1377433696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8140a7b20 0x7fb81413e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.434+0000 7fb81259c700 1 -- 172.21.15.135:0/1377433696 >> v2:172.21.15.40:3300/0 conn(0x7fb81413e710 msgr2=0x7fb814141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:52.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.434+0000 7fb81259c700 1 --2- 172.21.15.135:0/1377433696 >> v2:172.21.15.40:3300/0 conn(0x7fb81413e710 0x7fb814141760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.434+0000 7fb81259c700 10 monclient: _finish_hunting 0 2023-09-30T00:46:52.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.434+0000 7fb81259c700 1 monclient: found mon.noname-c 2023-09-30T00:46:52.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.434+0000 7fb81259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:52.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.434+0000 7fb81259c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:46:52.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.434+0000 7fb81259c700 1 -- 172.21.15.135:0/1377433696 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8140aef30 con 0x7fb8140a5710 2023-09-30T00:46:52.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.436+0000 7fb81259c700 1 --2- 172.21.15.135:0/1377433696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8140a5710 0x7fb8140a75e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb8140b1d90 tx=0x7fb8040091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16f5a65a5722bedd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:52.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.438+0000 7fb81159a700 1 -- 172.21.15.135:0/1377433696 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb804011630 con 0x7fb8140a5710 2023-09-30T00:46:52.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.438+0000 7fb81159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:52.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.438+0000 7fb81159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:52.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.438+0000 7fb81159a700 10 monclient: dump: 2023-09-30T00:46:52.441 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:52.441 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:52.441 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.441 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.441 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:52.441 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:52.441 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:52.441 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:52.441 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:52.441 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:52.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.438+0000 7fb81159a700 10 monclient: _finish_auth 0 2023-09-30T00:46:52.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.438+0000 7fb81159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:52.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.438+0000 7fb818a08700 5 monclient: authenticate success, global_id 4156 2023-09-30T00:46:52.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.438+0000 7fb818a08700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:52.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.439+0000 7fb81159a700 1 -- 172.21.15.135:0/1377433696 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8040117b0 con 0x7fb8140a5710 2023-09-30T00:46:52.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb81159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:52.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb81159a700 1 -- 172.21.15.135:0/1377433696 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb804011b70 con 0x7fb8140a5710 2023-09-30T00:46:52.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb818a08700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:52.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb818a08700 4 set_mon_vals no callback set 2023-09-30T00:46:52.447 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb81159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:52.447 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb81159a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:52.447 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb81159a700 10 monclient: dump: 2023-09-30T00:46:52.447 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:52.447 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:52.447 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.447 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.447 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:52.447 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:52.447 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:52.447 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:52.447 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:52.448 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:52.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb818a08700 10 monclient: shutdown 2023-09-30T00:46:52.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb818a08700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:52.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb818a08700 1 -- 172.21.15.135:0/1377433696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8140a5710 msgr2=0x7fb8140a75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:52.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb818a08700 1 --2- 172.21.15.135:0/1377433696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8140a5710 0x7fb8140a75e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb8140b1d90 tx=0x7fb8040091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb818a08700 1 -- 172.21.15.135:0/1377433696 shutdown_connections 2023-09-30T00:46:52.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb818a08700 1 --2- 172.21.15.135:0/1377433696 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8140a5710 0x7fb8140a75e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb818a08700 1 --2- 172.21.15.135:0/1377433696 >> v2:172.21.15.40:3300/0 conn(0x7fb81413e710 0x7fb814141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb818a08700 1 --2- 172.21.15.135:0/1377433696 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb8140a7b20 0x7fb81413e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb818a08700 1 -- 172.21.15.135:0/1377433696 >> 172.21.15.135:0/1377433696 conn(0x7fb814069080 msgr2=0x7fb8140631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:52.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb818a08700 1 -- 172.21.15.135:0/1377433696 shutdown_connections 2023-09-30T00:46:52.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.440+0000 7fb818a08700 1 -- 172.21.15.135:0/1377433696 wait complete. 2023-09-30T00:46:52.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.441+0000 7fb818a08700 10 monclient: build_initial_monmap 2023-09-30T00:46:52.449 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.441+0000 7fb818a08700 10 monclient: monmap: 2023-09-30T00:46:52.449 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2023-09-30T00:46:52.449 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:52.449 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2023-09-30T00:46:52.449 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2023-09-30T00:46:52.449 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:52.449 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2023-09-30T00:46:52.449 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:46:52.449 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:46:52.449 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:46:52.449 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:52.449 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.442+0000 7fb818a08700 1 librados: starting msgr at 2023-09-30T00:46:52.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.442+0000 7fb818a08700 1 librados: starting objecter 2023-09-30T00:46:52.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.442+0000 7fb818a08700 1 Processor -- start 2023-09-30T00:46:52.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.442+0000 7fb818a08700 1 -- start start 2023-09-30T00:46:52.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.442+0000 7fb818a08700 1 librados: setting wanted keys 2023-09-30T00:46:52.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.442+0000 7fb818a08700 1 librados: calling monclient init 2023-09-30T00:46:52.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.442+0000 7fb818a08700 10 monclient: init 2023-09-30T00:46:52.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.442+0000 7fb818a08700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:52.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.442+0000 7fb818a08700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:46:52.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.442+0000 7fb818a08700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8140a5710 0x7fb8140a90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.442+0000 7fb818a08700 10 monclient(hunting): picked mon.noname-a con 0x7fb8140a5710 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:46:52.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.442+0000 7fb818a08700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8140a7b20 0x7fb814151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.442+0000 7fb818a08700 10 monclient(hunting): picked mon.noname-b con 0x7fb8140a7b20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:46:52.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb818a08700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb81413e710 0x7fb814151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb818a08700 10 monclient(hunting): picked mon.noname-c con 0x7fb81413e710 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:52.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb818a08700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:52.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb818a08700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8140ab020 con 0x7fb8140a5710 2023-09-30T00:46:52.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb818a08700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:52.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb818a08700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81413d950 con 0x7fb8140a7b20 2023-09-30T00:46:52.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb818a08700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:52.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb818a08700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81413dac0 con 0x7fb81413e710 2023-09-30T00:46:52.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb818a08700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:52.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb818a08700 10 monclient(hunting): authenticate will time out at 5280.882047s 2023-09-30T00:46:52.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb811d9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8140a7b20 0x7fb814151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb81259c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8140a5710 0x7fb8140a90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb812d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb81413e710 0x7fb814151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb811d9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8140a7b20 0x7fb814151450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.135:54246/0 (socket says 172.21.15.135:54246) 2023-09-30T00:46:52.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb811d9b700 1 -- 172.21.15.135:0/1863860230 learned_addr learned my addr 172.21.15.135:0/1863860230 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-09-30T00:46:52.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb811d9b700 10 monclient(hunting): get_auth_request con 0x7fb8140a7b20 auth_method 0 2023-09-30T00:46:52.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb811d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:52.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb811d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:52.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb811d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:52.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb81259c700 10 monclient(hunting): get_auth_request con 0x7fb8140a5710 auth_method 0 2023-09-30T00:46:52.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb81259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:52.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb81259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:52.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.443+0000 7fb81259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:52.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.444+0000 7fb812d9d700 10 monclient(hunting): get_auth_request con 0x7fb81413e710 auth_method 0 2023-09-30T00:46:52.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.444+0000 7fb812d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:52.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.444+0000 7fb812d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:52.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.444+0000 7fb812d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:52.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.444+0000 7fb811d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:52.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.444+0000 7fb811d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:52.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.444+0000 7fb811d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:52.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.444+0000 7fb81259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:52.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.444+0000 7fb81259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:52.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.444+0000 7fb81259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:52.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.444+0000 7fb812d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:52.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.444+0000 7fb812d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:52.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.444+0000 7fb812d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:52.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.447+0000 7fb811d9b700 10 monclient(hunting): handle_auth_done global_id 4159 payload 824 2023-09-30T00:46:52.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.447+0000 7fb811d9b700 1 -- 172.21.15.135:0/1863860230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb81413e710 msgr2=0x7fb814151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:52.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.447+0000 7fb811d9b700 1 --2- 172.21.15.135:0/1863860230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb81413e710 0x7fb814151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.447+0000 7fb811d9b700 1 -- 172.21.15.135:0/1863860230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8140a5710 msgr2=0x7fb8140a90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:52.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.447+0000 7fb811d9b700 1 --2- 172.21.15.135:0/1863860230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8140a5710 0x7fb8140a90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.447+0000 7fb811d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:46:52.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.447+0000 7fb811d9b700 1 monclient: found mon.noname-b 2023-09-30T00:46:52.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.447+0000 7fb811d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:52.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.447+0000 7fb811d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:46:52.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.447+0000 7fb811d9b700 1 -- 172.21.15.135:0/1863860230 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb804009420 con 0x7fb8140a7b20 2023-09-30T00:46:52.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.449+0000 7fb811d9b700 1 --2- 172.21.15.135:0/1863860230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8140a7b20 0x7fb814151450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb80800ec80 tx=0x7fb8080091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:52.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.449+0000 7fb8027fc700 10 client.?.objecter ms_handle_connect 0x7fb8140a7b20 2023-09-30T00:46:52.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.449+0000 7fb8027fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:46:52.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb8027fc700 1 -- 172.21.15.135:0/1863860230 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80800c850 con 0x7fb8140a7b20 2023-09-30T00:46:52.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb8027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:52.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb8027fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:52.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb8027fc700 10 monclient: dump: 2023-09-30T00:46:52.457 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:52.457 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:52.457 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.457 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.457 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:52.457 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:52.457 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:52.457 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:52.457 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:52.457 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:52.457 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb8027fc700 10 monclient: _finish_auth 0 2023-09-30T00:46:52.457 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb8027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:52.457 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb8027fc700 1 -- 172.21.15.135:0/1863860230 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80800c9b0 con 0x7fb8140a7b20 2023-09-30T00:46:52.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb818a08700 5 monclient: authenticate success, global_id 4159 2023-09-30T00:46:52.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb818a08700 10 monclient: _renew_subs 2023-09-30T00:46:52.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb818a08700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:52.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb818a08700 1 -- 172.21.15.135:0/1863860230 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8140a85f0 con 0x7fb8140a7b20 2023-09-30T00:46:52.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb8027fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:52.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb8027fc700 1 -- 172.21.15.135:0/1863860230 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80800cd50 con 0x7fb8140a7b20 2023-09-30T00:46:52.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb8027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:52.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb8027fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:52.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb8027fc700 10 monclient: dump: 2023-09-30T00:46:52.458 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2023-09-30T00:46:52.458 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:52.458 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.459 INFO:teuthology.orchestra.run.smithi135.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:52.459 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:52.459 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 3 2023-09-30T00:46:52.459 INFO:teuthology.orchestra.run.smithi135.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:52.459 INFO:teuthology.orchestra.run.smithi135.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:52.459 INFO:teuthology.orchestra.run.smithi135.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:52.459 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-09-30T00:46:52.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb818a08700 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:46:52.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb818a08700 10 monclient: _renew_subs 2023-09-30T00:46:52.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb818a08700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:52.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb810d99700 4 set_mon_vals no callback set 2023-09-30T00:46:52.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb818a08700 1 -- 172.21.15.135:0/1863860230 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb814152090 con 0x7fb8140a7b20 2023-09-30T00:46:52.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.453+0000 7fb818a08700 1 librados: init done 2023-09-30T00:46:52.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.455+0000 7fb818a08700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:52.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.455+0000 7fb818a08700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:52.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.455+0000 7fb818a08700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:52.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.455+0000 7fb818a08700 1 -- 172.21.15.135:0/1863860230 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb814072b90 con 0x7fb8140a7b20 2023-09-30T00:46:52.520 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:52.519+0000 7feafc14a1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T00:46:52.559 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.558+0000 7fb8027fc700 1 -- 172.21.15.135:0/1863860230 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fb808005c80 con 0x7fb8140a7b20 2023-09-30T00:46:52.560 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.558+0000 7fb8027fc700 1 -- 172.21.15.135:0/1863860230 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7fb808006640 con 0x7fb8140a7b20 2023-09-30T00:46:52.560 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.558+0000 7fb8027fc700 10 client.4159.objecter ms_dispatch 0x7fb814001550 osd_map(9..9 src has 1..9) v4 2023-09-30T00:46:52.560 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.558+0000 7fb8027fc700 3 client.4159.objecter handle_osd_map got epochs [9,9] > 0 2023-09-30T00:46:52.560 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.558+0000 7fb8027fc700 3 client.4159.objecter handle_osd_map decoding full epoch 9 2023-09-30T00:46:52.560 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.558+0000 7fb8027fc700 20 client.4159.objecter dump_active .. 0 homeless 2023-09-30T00:46:52.560 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.558+0000 7fb8027fc700 1 -- 172.21.15.135:0/1863860230 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fb808021060 con 0x7fb8140a7b20 2023-09-30T00:46:52.561 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.558+0000 7fb8027fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:52.561 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.558+0000 7fb8027fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:52.588 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:52.587+0000 7fadf12501c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T00:46:52.642 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.640+0000 7fb818a08700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7", "id": 7}] 2023-09-30T00:46:52.642 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.640+0000 7fb818a08700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7", "id": 7}] 2023-09-30T00:46:52.642 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.640+0000 7fb818a08700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:46:52.642 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.640+0000 7fb818a08700 1 -- 172.21.15.135:0/1863860230 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7", "id": 7} v 0) v1 -- 0x7fb8140b1b00 con 0x7fb8140a7b20 2023-09-30T00:46:52.667 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:52.666+0000 7feafc14a1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:46:52.773 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.771+0000 7fb8027fc700 1 -- 172.21.15.135:0/1863860230 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb808017070 con 0x7fb8140a7b20 2023-09-30T00:46:52.773 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.771+0000 7fb8027fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7", "id": 7}] 2023-09-30T00:46:52.773 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.771+0000 7fb8027fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:52.773 INFO:teuthology.orchestra.run.smithi135.stdout:7 2023-09-30T00:46:52.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.772+0000 7fb818a08700 10 librados: watch_flush enter 2023-09-30T00:46:52.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.772+0000 7fb818a08700 10 librados: watch_flush exit 2023-09-30T00:46:52.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.772+0000 7fb818a08700 20 client.4159.objecter shutdown clearing up homeless session... 2023-09-30T00:46:52.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.772+0000 7fb818a08700 10 client.4159.objecter successfully canceled tick 2023-09-30T00:46:52.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.772+0000 7fb818a08700 10 monclient: shutdown 2023-09-30T00:46:52.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.772+0000 7fb818a08700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:52.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.772+0000 7fb818a08700 1 -- 172.21.15.135:0/1863860230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8140a7b20 msgr2=0x7fb814151450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:52.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.772+0000 7fb818a08700 1 --2- 172.21.15.135:0/1863860230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8140a7b20 0x7fb814151450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb80800ec80 tx=0x7fb8080091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.772+0000 7fb818a08700 1 -- 172.21.15.135:0/1863860230 shutdown_connections 2023-09-30T00:46:52.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.772+0000 7fb818a08700 1 --2- 172.21.15.135:0/1863860230 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb81413e710 0x7fb814151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.772+0000 7fb818a08700 1 --2- 172.21.15.135:0/1863860230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8140a5710 0x7fb8140a90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.775 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.772+0000 7fb818a08700 1 --2- 172.21.15.135:0/1863860230 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb8140a7b20 0x7fb814151450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.776 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.772+0000 7fb818a08700 1 -- 172.21.15.135:0/1863860230 >> 172.21.15.135:0/1863860230 conn(0x7fb814069080 msgr2=0x7fb814138260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:52.776 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.773+0000 7fb818a08700 1 -- 172.21.15.135:0/1863860230 shutdown_connections 2023-09-30T00:46:52.776 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.773+0000 7fb818a08700 1 -- 172.21.15.135:0/1863860230 wait complete. 2023-09-30T00:46:52.776 INFO:teuthology.orchestra.run.smithi135.stderr:2023-09-30T00:46:52.773+0000 7fb818a08700 1 librados: shutdown 2023-09-30T00:46:52.785 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T00:46:52.785 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-30T00:46:52.786 DEBUG:teuthology.orchestra.run.smithi040:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-09-30T00:46:52.788 INFO:tasks.ceph.osd.0:Started 2023-09-30T00:46:52.788 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T00:46:52.788 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-30T00:46:52.788 DEBUG:teuthology.orchestra.run.smithi040:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-09-30T00:46:52.790 INFO:tasks.ceph.osd.1:Started 2023-09-30T00:46:52.790 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T00:46:52.791 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-30T00:46:52.791 DEBUG:teuthology.orchestra.run.smithi040:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-09-30T00:46:52.793 INFO:tasks.ceph.osd.2:Started 2023-09-30T00:46:52.793 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T00:46:52.793 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-30T00:46:52.793 DEBUG:teuthology.orchestra.run.smithi040:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-09-30T00:46:52.795 INFO:tasks.ceph.osd.3:Started 2023-09-30T00:46:52.795 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T00:46:52.795 INFO:tasks.ceph.osd.4:Restarting daemon 2023-09-30T00:46:52.796 DEBUG:teuthology.orchestra.run.smithi135:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-09-30T00:46:52.798 INFO:tasks.ceph.osd.4:Started 2023-09-30T00:46:52.798 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T00:46:52.798 INFO:tasks.ceph.osd.5:Restarting daemon 2023-09-30T00:46:52.798 DEBUG:teuthology.orchestra.run.smithi135:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-09-30T00:46:52.800 INFO:tasks.ceph.osd.5:Started 2023-09-30T00:46:52.800 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T00:46:52.800 INFO:tasks.ceph.osd.6:Restarting daemon 2023-09-30T00:46:52.800 DEBUG:teuthology.orchestra.run.smithi135:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-09-30T00:46:52.802 INFO:tasks.ceph.osd.6:Started 2023-09-30T00:46:52.802 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T00:46:52.802 INFO:tasks.ceph.osd.7:Restarting daemon 2023-09-30T00:46:52.803 DEBUG:teuthology.orchestra.run.smithi135:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-09-30T00:46:52.804 INFO:tasks.ceph.osd.7:Started 2023-09-30T00:46:52.805 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:46:52.826 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:52.825+0000 7feafc14a1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T00:46:52.959 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:52.958+0000 7feafc14a1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T00:46:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.020+0000 7f93ba0d3700 10 monclient: get_monmap_and_config 2023-09-30T00:46:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.020+0000 7f93ba0d3700 10 monclient: build_initial_monmap 2023-09-30T00:46:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.020+0000 7f93ba0d3700 10 monclient: monmap: 2023-09-30T00:46:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:46:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:46:53.021627+0000 2023-09-30T00:46:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:46:53.021627+0000 2023-09-30T00:46:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:46:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:46:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:46:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:46:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:46:53.022 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:46:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.021+0000 7f93ba0d3700 1 Processor -- start 2023-09-30T00:46:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.021+0000 7f93ba0d3700 1 -- start start 2023-09-30T00:46:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.021+0000 7f93ba0d3700 10 monclient: init 2023-09-30T00:46:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.022+0000 7f93ba0d3700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.022+0000 7f93ba0d3700 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-09-30T00:46:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.022+0000 7f93ba0d3700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93b40b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:46:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.022+0000 7f93ba0d3700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f93b40b1830 0x7f93b40a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.022+0000 7f93ba0d3700 10 monclient(hunting): picked mon.noname-c con 0x7f93b40b1830 addr v2:172.21.15.135:3300/0 2023-09-30T00:46:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.022+0000 7f93ba0d3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93b413ac20 addr v1:172.21.15.135:6789/0 2023-09-30T00:46:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.022+0000 7f93ba0d3700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b40a5b20 con 0x7f93b40b1460 2023-09-30T00:46:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.022+0000 7f93ba0d3700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b40a71f0 con 0x7f93b413ac20 2023-09-30T00:46:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.022+0000 7f93ba0d3700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.022+0000 7f93ba0d3700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b4141e40 con 0x7f93b40b1830 2023-09-30T00:46:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.022+0000 7f93ba0d3700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.022+0000 7f93ba0d3700 10 monclient(hunting): authenticate will time out at 5274.323874s 2023-09-30T00:46:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.023+0000 7f93b37fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f93b40b1460 0x7f93b4141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34856/0 (socket says 172.21.15.40:34856) 2023-09-30T00:46:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.023+0000 7f93b37fe700 1 -- 172.21.15.40:0/995296203 learned_addr learned my addr 172.21.15.40:0/995296203 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:46:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.028+0000 7f93b27fc700 1 -- 172.21.15.40:0/995296203 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666470300 0 0) 0x7f93b40a5b20 con 0x7f93b40b1460 2023-09-30T00:46:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.028+0000 7f93b27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.028+0000 7f93b27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.028+0000 7f93b27fc700 10 monclient(hunting): my global_id is 4164 2023-09-30T00:46:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.028+0000 7f93b27fc700 1 -- 172.21.15.40:0/995296203 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93a0003570 con 0x7f93b40b1460 2023-09-30T00:46:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.035+0000 7f93b27fc700 1 -- 172.21.15.40:0/995296203 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2014054061 0 0) 0x7f93a0003570 con 0x7f93b40b1460 2023-09-30T00:46:53.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 1 -- 172.21.15.40:0/995296203 >> v2:172.21.15.135:3300/0 conn(0x7f93b40b1830 msgr2=0x7f93b40a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:53.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 1 --2- 172.21.15.40:0/995296203 >> v2:172.21.15.135:3300/0 conn(0x7f93b40b1830 0x7f93b40a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:53.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 1 -- 172.21.15.40:0/995296203 >> v1:172.21.15.135:6789/0 conn(0x7f93b413ac20 legacy=0x7f93b413b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:53.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient: _finish_hunting 0 2023-09-30T00:46:53.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 1 monclient: found mon.noname-a-legacy 2023-09-30T00:46:53.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:53.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2023-09-30T00:46:53.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 1 -- 172.21.15.40:0/995296203 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b40a74d0 con 0x7f93b40b1460 2023-09-30T00:46:53.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient: _finish_auth 0 2023-09-30T00:46:53.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:53.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 1 -- 172.21.15.40:0/995296203 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3033000540 0 0) 0x7f939c001470 con 0x7f93b40b1460 2023-09-30T00:46:53.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:53.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T00:46:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient: dump: 2023-09-30T00:46:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:46:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:53.038 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:46:53.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2023-09-30T00:46:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 1 -- 172.21.15.40:0/995296203 >> v1:172.21.15.40:6789/0 conn(0x7f93b40b1460 legacy=0x7f93b4141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:46:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 1 --2- 172.21.15.40:0/995296203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93a0003ae0 0x7f93a0003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient(hunting): picked mon.a con 0x7f93a0003ae0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:46:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 1 --2- 172.21.15.40:0/995296203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93a0004810 0x7f93a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient(hunting): picked mon.b con 0x7f93a0004810 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:46:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 1 --2- 172.21.15.40:0/995296203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a00053c0 0x7f93a000ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient(hunting): picked mon.c con 0x7f93a00053c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 1 -- 172.21.15.40:0/995296203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b4141e40 con 0x7f93a0003ae0 2023-09-30T00:46:53.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 1 -- 172.21.15.40:0/995296203 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a000c360 con 0x7f93a0004810 2023-09-30T00:46:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 1 -- 172.21.15.40:0/995296203 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a000c520 con 0x7f93a00053c0 2023-09-30T00:46:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:53.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.036+0000 7f93b27fc700 10 monclient(hunting): _finish_auth 0 2023-09-30T00:46:53.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.037+0000 7f93b3fff700 1 --2- 172.21.15.40:0/995296203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a00053c0 0x7f93a000ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:53.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.037+0000 7f93b3fff700 10 monclient(hunting): get_auth_request con 0x7f93a00053c0 auth_method 0 2023-09-30T00:46:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.037+0000 7f93b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.037+0000 7f93b3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.037+0000 7f93b3fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.038+0000 7f93b3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.038+0000 7f93b3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.038+0000 7f93b3fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:46:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.042+0000 7f93b37fe700 1 --2- 172.21.15.40:0/995296203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93a0003ae0 0x7f93a0003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.042+0000 7f93b37fe700 10 monclient(hunting): get_auth_request con 0x7f93a0003ae0 auth_method 0 2023-09-30T00:46:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.042+0000 7f93b37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.042+0000 7f93b37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:53.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.042+0000 7f93b37fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:46:53.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:53.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:53.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b37fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:46:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b3fff700 10 monclient(hunting): handle_auth_done global_id 4164 payload 293 2023-09-30T00:46:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b3fff700 1 -- 172.21.15.40:0/995296203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93a0004810 msgr2=0x7f93a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b3fff700 1 --2- 172.21.15.40:0/995296203 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93a0004810 0x7f93a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b3fff700 1 -- 172.21.15.40:0/995296203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93a0003ae0 msgr2=0x7f93a0003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b3fff700 1 --2- 172.21.15.40:0/995296203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93a0003ae0 0x7f93a0003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b3fff700 10 monclient: _finish_hunting 0 2023-09-30T00:46:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b3fff700 1 monclient: found mon.c 2023-09-30T00:46:53.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:53.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b3fff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:46:53.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b3fff700 1 -- 172.21.15.40:0/995296203 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b40a74d0 con 0x7f93a00053c0 2023-09-30T00:46:53.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b3fff700 10 monclient: _finish_auth 0 2023-09-30T00:46:53.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:53.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93b3fff700 1 -- 172.21.15.40:0/995296203 reap_dead start 2023-09-30T00:46:53.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93ba0d3700 5 monclient: authenticate success, global_id 4164 2023-09-30T00:46:53.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.043+0000 7f93ba0d3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:53.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.046+0000 7f93b3fff700 1 --2- 172.21.15.40:0/995296203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a00053c0 0x7f93a000ba80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f93ac001ec0 tx=0x7f93ac0014d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f71e44d7ec16176 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:53.047 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:53.046+0000 7fadf12501c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T00:46:53.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.053+0000 7f93b27fc700 1 -- 172.21.15.40:0/995296203 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac001800 con 0x7f93a00053c0 2023-09-30T00:46:53.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.053+0000 7f93b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.053+0000 7f93b27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.053+0000 7f93b27fc700 10 monclient: dump: 2023-09-30T00:46:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:46:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:53.055 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:46:53.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.053+0000 7f93b27fc700 1 -- 172.21.15.40:0/995296203 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac00c6a0 con 0x7f93a00053c0 2023-09-30T00:46:53.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.053+0000 7f93b27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:53.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.053+0000 7f93b27fc700 1 -- 172.21.15.40:0/995296203 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac00ca60 con 0x7f93a00053c0 2023-09-30T00:46:53.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.053+0000 7f93b27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:53.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.053+0000 7f93b27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:53.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.053+0000 7f93b27fc700 10 monclient: dump: 2023-09-30T00:46:53.055 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:46:53.055 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:53.055 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:53.055 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:53.055 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:53.056 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:53.056 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:53.056 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:53.056 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:53.056 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:46:53.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.053+0000 7f93ba0d3700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:53.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.053+0000 7f93ba0d3700 4 set_mon_vals no callback set 2023-09-30T00:46:53.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.054+0000 7f93ba0d3700 10 monclient: shutdown 2023-09-30T00:46:53.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.054+0000 7f93ba0d3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:53.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.054+0000 7f93ba0d3700 1 -- 172.21.15.40:0/995296203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a00053c0 msgr2=0x7f93a000ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:53.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.054+0000 7f93ba0d3700 1 --2- 172.21.15.40:0/995296203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a00053c0 0x7f93a000ba80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f93ac001ec0 tx=0x7f93ac0014d0 comp rx=0 tx=0).stop 2023-09-30T00:46:53.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.054+0000 7f93ba0d3700 1 -- 172.21.15.40:0/995296203 shutdown_connections 2023-09-30T00:46:53.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.054+0000 7f93ba0d3700 1 --2- 172.21.15.40:0/995296203 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a00053c0 0x7f93a000ba80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:53.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.054+0000 7f93ba0d3700 1 -- 172.21.15.40:0/995296203 >> 172.21.15.40:0/995296203 conn(0x7f93b40631f0 msgr2=0x7f93b40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:53.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.056+0000 7f93ba0d3700 1 -- 172.21.15.40:0/995296203 shutdown_connections 2023-09-30T00:46:53.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.057+0000 7f93ba0d3700 1 -- 172.21.15.40:0/995296203 wait complete. 2023-09-30T00:46:53.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.058+0000 7f93ba0d3700 10 monclient: build_initial_monmap 2023-09-30T00:46:53.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.058+0000 7f93ba0d3700 10 monclient: monmap: 2023-09-30T00:46:53.068 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:46:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:46:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:46:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:46:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:46:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:46:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:46:53.069 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:46:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.058+0000 7f93ba0d3700 1 librados: starting msgr at 2023-09-30T00:46:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.058+0000 7f93ba0d3700 1 librados: starting objecter 2023-09-30T00:46:53.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.058+0000 7f93ba0d3700 1 Processor -- start 2023-09-30T00:46:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.059+0000 7f93ba0d3700 1 -- start start 2023-09-30T00:46:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.059+0000 7f93ba0d3700 1 librados: setting wanted keys 2023-09-30T00:46:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.059+0000 7f93ba0d3700 1 librados: calling monclient init 2023-09-30T00:46:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.059+0000 7f93ba0d3700 10 monclient: init 2023-09-30T00:46:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.059+0000 7f93ba0d3700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.059+0000 7f93ba0d3700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:46:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.059+0000 7f93ba0d3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a00053c0 0x7f93b415d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.060+0000 7f93ba0d3700 10 monclient(hunting): picked mon.noname-c con 0x7f93a00053c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.060+0000 7f93ba0d3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93b415e410 0x7f93b40a8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.060+0000 7f93ba0d3700 10 monclient(hunting): picked mon.noname-b con 0x7f93b415e410 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:46:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.060+0000 7f93ba0d3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93b406c9a0 0x7f93b40a8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:53.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.060+0000 7f93ba0d3700 10 monclient(hunting): picked mon.noname-a con 0x7f93b406c9a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:46:53.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.060+0000 7f93ba0d3700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:53.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.060+0000 7f93ba0d3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b40a88f0 con 0x7f93b406c9a0 2023-09-30T00:46:53.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.060+0000 7f93ba0d3700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:53.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.060+0000 7f93ba0d3700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b40a8540 con 0x7f93b415e410 2023-09-30T00:46:53.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.060+0000 7f93ba0d3700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:53.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.060+0000 7f93ba0d3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b40a92c0 con 0x7f93a00053c0 2023-09-30T00:46:53.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.060+0000 7f93ba0d3700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:53.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.060+0000 7f93ba0d3700 10 monclient(hunting): authenticate will time out at 5274.361079s 2023-09-30T00:46:53.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.061+0000 7f93b37fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a00053c0 0x7f93b415d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:53.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.061+0000 7f93b37fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a00053c0 0x7f93b415d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:57412/0 (socket says 172.21.15.40:57412) 2023-09-30T00:46:53.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.061+0000 7f93b37fe700 1 -- 172.21.15.40:0/1269710390 learned_addr learned my addr 172.21.15.40:0/1269710390 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:46:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.061+0000 7f93b37fe700 10 monclient(hunting): get_auth_request con 0x7f93a00053c0 auth_method 0 2023-09-30T00:46:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.061+0000 7f93b37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.061+0000 7f93b37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.061+0000 7f93b37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.062+0000 7f93b37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.062+0000 7f93b37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.062+0000 7f93b37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.067+0000 7f93b37fe700 10 monclient(hunting): handle_auth_done global_id 4142 payload 824 2023-09-30T00:46:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.067+0000 7f93b37fe700 1 -- 172.21.15.40:0/1269710390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93b415e410 msgr2=0x7f93b40a8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.067+0000 7f93b37fe700 1 --2- 172.21.15.40:0/1269710390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93b415e410 0x7f93b40a8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.067+0000 7f93b37fe700 1 -- 172.21.15.40:0/1269710390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93b406c9a0 msgr2=0x7f93b40a8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.067+0000 7f93b37fe700 1 --2- 172.21.15.40:0/1269710390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93b406c9a0 0x7f93b40a8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:53.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.067+0000 7f93b37fe700 10 monclient: _finish_hunting 0 2023-09-30T00:46:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.067+0000 7f93b37fe700 1 monclient: found mon.noname-c 2023-09-30T00:46:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.067+0000 7f93b37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.067+0000 7f93b37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:46:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.067+0000 7f93b37fe700 1 -- 172.21.15.40:0/1269710390 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b40a9480 con 0x7f93a00053c0 2023-09-30T00:46:53.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.073+0000 7f93b37fe700 1 --2- 172.21.15.40:0/1269710390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a00053c0 0x7f93b415d920 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f939c007510 tx=0x7f939c001e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:53.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.073+0000 7f93ab7fe700 10 client.?.objecter ms_handle_connect 0x7f93a00053c0 2023-09-30T00:46:53.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.073+0000 7f93ab7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:46:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.079+0000 7f93ab7fe700 1 -- 172.21.15.40:0/1269710390 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c018030 con 0x7f93a00053c0 2023-09-30T00:46:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.079+0000 7f93ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.079+0000 7f93ab7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.079+0000 7f93ab7fe700 10 monclient: dump: 2023-09-30T00:46:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:46:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:53.082 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:46:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ab7fe700 10 monclient: _finish_auth 0 2023-09-30T00:46:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ab7fe700 1 -- 172.21.15.40:0/1269710390 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939c00f9f0 con 0x7f93a00053c0 2023-09-30T00:46:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ab7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ab7fe700 1 -- 172.21.15.40:0/1269710390 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c00fdb0 con 0x7f93a00053c0 2023-09-30T00:46:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ab7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ab7fe700 10 monclient: dump: 2023-09-30T00:46:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:46:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:53.083 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:46:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93b17fa700 4 set_mon_vals no callback set 2023-09-30T00:46:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ba0d3700 5 monclient: authenticate success, global_id 4142 2023-09-30T00:46:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ba0d3700 10 monclient: _renew_subs 2023-09-30T00:46:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ba0d3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:46:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ba0d3700 1 -- 172.21.15.40:0/1269710390 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b415a1d0 con 0x7f93a00053c0 2023-09-30T00:46:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ba0d3700 10 client.4142.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:46:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ba0d3700 10 monclient: _renew_subs 2023-09-30T00:46:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ba0d3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:46:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ba0d3700 1 -- 172.21.15.40:0/1269710390 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b415a670 con 0x7f93a00053c0 2023-09-30T00:46:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.080+0000 7f93ba0d3700 1 librados: init done 2023-09-30T00:46:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.083+0000 7f93ba0d3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.083+0000 7f93ba0d3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.083+0000 7f93ba0d3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:46:53.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.083+0000 7f93ba0d3700 1 -- 172.21.15.40:0/1269710390 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b40a8700 con 0x7f93a00053c0 2023-09-30T00:46:53.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.122+0000 7f93ab7fe700 1 -- 172.21.15.40:0/1269710390 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f939c01b070 con 0x7f93a00053c0 2023-09-30T00:46:53.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.122+0000 7f93ab7fe700 1 -- 172.21.15.40:0/1269710390 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f939c019070 con 0x7f93a00053c0 2023-09-30T00:46:53.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.122+0000 7f93ab7fe700 10 client.4142.objecter ms_dispatch 0x7f93b4001550 osd_map(10..10 src has 1..10) v4 2023-09-30T00:46:53.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.122+0000 7f93ab7fe700 3 client.4142.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T00:46:53.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.122+0000 7f93ab7fe700 3 client.4142.objecter handle_osd_map decoding full epoch 10 2023-09-30T00:46:53.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.122+0000 7f93ab7fe700 20 client.4142.objecter dump_active .. 0 homeless 2023-09-30T00:46:53.327 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:53.326+0000 7fadf12501c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:46:53.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.495+0000 7f93ab7fe700 1 -- 172.21.15.40:0/1269710390 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f939c013600 con 0x7f93a00053c0 2023-09-30T00:46:53.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.495+0000 7f93ab7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:53.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.495+0000 7f93ab7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.642+0000 7f93ba0d3700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:46:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.642+0000 7f93ba0d3700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:46:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.642+0000 7f93ba0d3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:46:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.642+0000 7f93ba0d3700 1 -- 172.21.15.40:0/1269710390 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93b4064a00 con 0x7f93a00053c0 2023-09-30T00:46:53.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:53.768+0000 7f93ab7fe700 1 -- 172.21.15.40:0/1269710390 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mgrmap(e 2) v1 ==== 42252+0+0 (secure 0 0 0) 0x7f939c013bd0 con 0x7f93a00053c0 2023-09-30T00:46:54.056 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:54.055+0000 7fadf12501c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T00:46:54.350 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:54.347+0000 7fadf12501c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:46:54.423 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:54.422+0000 7fea8b5f4700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:46:54.423 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:54.422+0000 7fea8b5f4700 -1 pg_autoscaler.serve: 2023-09-30T00:46:54.425 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:54.422+0000 7fea8b5f4700 -1 Traceback (most recent call last): 2023-09-30T00:46:54.425 INFO:tasks.ceph.mgr.x.smithi135.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-30T00:46:54.425 INFO:tasks.ceph.mgr.x.smithi135.stderr: self._maybe_adjust() 2023-09-30T00:46:54.426 INFO:tasks.ceph.mgr.x.smithi135.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-30T00:46:54.426 INFO:tasks.ceph.mgr.x.smithi135.stderr: if self.noautoscale: 2023-09-30T00:46:54.426 INFO:tasks.ceph.mgr.x.smithi135.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:46:54.426 INFO:tasks.ceph.mgr.x.smithi135.stderr: 2023-09-30T00:46:54.650 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:54.649+0000 7fadf12501c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T00:46:54.653 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:46:54.653 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":10,"fsid":"238dba7c-9190-48bd-bb67-322ebc9ecbd0","created":"2023-09-30T00:46:45.822318+0000","modified":"2023-09-30T00:46:52.657791+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:46:52.657791+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"66e1da65-310e-4463-8aa0-431896832e64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d6dd269f-2eda-4ef7-9da1-2ed5b16291b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f0de27d-3674-4ff4-8451-4fcc90786c98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d18ec2f5-7e8f-4732-a523-adc495115cbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e2465fe-bb27-47bd-8ddf-f334cac6041a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7ea449db-9b22-48b8-a61b-1937eeeb6c8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c79b4829-6393-4f9a-a9f8-d2aacb505ae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:46:54.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.650+0000 7f93ab7fe700 1 -- 172.21.15.40:0/1269710390 <== mon.2 v2:172.21.15.40:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f939c01b6e0 con 0x7f93a00053c0 2023-09-30T00:46:54.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.650+0000 7f93ab7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:46:54.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.650+0000 7f93ab7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:54.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.652+0000 7f93ba0d3700 10 librados: watch_flush enter 2023-09-30T00:46:54.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.652+0000 7f93ba0d3700 10 librados: watch_flush exit 2023-09-30T00:46:54.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.652+0000 7f93ba0d3700 20 client.4142.objecter shutdown clearing up homeless session... 2023-09-30T00:46:54.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.652+0000 7f93ba0d3700 10 client.4142.objecter successfully canceled tick 2023-09-30T00:46:54.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.652+0000 7f93ba0d3700 10 monclient: shutdown 2023-09-30T00:46:54.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.652+0000 7f93ba0d3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:54.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.652+0000 7f93ba0d3700 1 -- 172.21.15.40:0/1269710390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a00053c0 msgr2=0x7f93b415d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:54.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.652+0000 7f93ba0d3700 1 --2- 172.21.15.40:0/1269710390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a00053c0 0x7f93b415d920 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f939c007510 tx=0x7f939c001e60 comp rx=0 tx=0).stop 2023-09-30T00:46:54.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.654+0000 7f93ba0d3700 1 -- 172.21.15.40:0/1269710390 shutdown_connections 2023-09-30T00:46:54.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.654+0000 7f93ba0d3700 1 --2- 172.21.15.40:0/1269710390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93b406c9a0 0x7f93b40a8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:54.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.654+0000 7f93ba0d3700 1 --2- 172.21.15.40:0/1269710390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93a00053c0 0x7f93b415d920 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:54.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.654+0000 7f93ba0d3700 1 --2- 172.21.15.40:0/1269710390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93b415e410 0x7f93b40a8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:54.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.654+0000 7f93ba0d3700 1 -- 172.21.15.40:0/1269710390 >> 172.21.15.40:0/1269710390 conn(0x7f93b40631f0 msgr2=0x7f93b40635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:54.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.658+0000 7f93ba0d3700 1 -- 172.21.15.40:0/1269710390 shutdown_connections 2023-09-30T00:46:54.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.661+0000 7f93ba0d3700 1 -- 172.21.15.40:0/1269710390 wait complete. 2023-09-30T00:46:54.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:46:54.661+0000 7f93ba0d3700 1 librados: shutdown 2023-09-30T00:46:54.685 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-30T00:46:54.685 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-30T00:46:54.850 INFO:tasks.ceph.mgr.y.smithi040.stderr:2023-09-30T00:46:54.850+0000 7fadf12501c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T00:46:55.491 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:46:55.692 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.691+0000 7f312c645700 10 monclient: get_monmap_and_config 2023-09-30T00:46:55.693 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.691+0000 7f312c645700 10 monclient: build_initial_monmap 2023-09-30T00:46:55.693 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.691+0000 7f312c645700 10 monclient: monmap: 2023-09-30T00:46:55.693 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:46:55.693 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:55.693 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:46:55.692874+0000 2023-09-30T00:46:55.693 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:46:55.692874+0000 2023-09-30T00:46:55.693 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:55.693 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:55.693 INFO:teuthology.misc.health.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:46:55.693 INFO:teuthology.misc.health.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:46:55.693 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:46:55.694 INFO:teuthology.misc.health.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:46:55.694 INFO:teuthology.misc.health.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:46:55.694 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:46:55.694 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.692+0000 7f312c645700 1 Processor -- start 2023-09-30T00:46:55.694 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.693+0000 7f312c645700 1 -- start start 2023-09-30T00:46:55.694 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.693+0000 7f312c645700 10 monclient: init 2023-09-30T00:46:55.695 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.694+0000 7f312c645700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:55.696 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.694+0000 7f312c645700 10 monclient: _add_conns ranks=[4,1,0,3,2] 2023-09-30T00:46:55.696 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.695+0000 7f312c645700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f31240b1980 0x7f31240afa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:55.696 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.695+0000 7f312c645700 10 monclient(hunting): picked mon.noname-c con 0x7f31240b1980 addr v2:172.21.15.135:3300/0 2023-09-30T00:46:55.696 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.695+0000 7f312c645700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3124145dc0 addr v1:172.21.15.135:6789/0 2023-09-30T00:46:55.696 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.695+0000 7f312c645700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f312406da60 addr v1:172.21.15.40:6789/0 2023-09-30T00:46:55.696 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.695+0000 7f312c645700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3124146950 con 0x7f312406da60 2023-09-30T00:46:55.696 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.695+0000 7f312c645700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31240b1d50 con 0x7f3124145dc0 2023-09-30T00:46:55.696 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.695+0000 7f312c645700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:55.696 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.695+0000 7f312c645700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31240b0040 con 0x7f31240b1980 2023-09-30T00:46:55.696 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.695+0000 7f312c645700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:55.696 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.695+0000 7f312c645700 10 monclient(hunting): authenticate will time out at 5276.996132s 2023-09-30T00:46:55.697 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.696+0000 7f312a3e1700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f31240b1980 0x7f31240afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:55.697 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.696+0000 7f3129be0700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f3124145dc0 0x7f3124146240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:35974/0 (socket says 172.21.15.40:35974) 2023-09-30T00:46:55.698 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.696+0000 7f3129be0700 1 -- 172.21.15.40:0/1456693109 learned_addr learned my addr 172.21.15.40:0/1456693109 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:46:55.698 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.696+0000 7f312a3e1700 10 monclient(hunting): get_auth_request con 0x7f31240b1980 auth_method 0 2023-09-30T00:46:55.698 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.696+0000 7f312a3e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:55.698 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.696+0000 7f312a3e1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:55.698 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.696+0000 7f312a3e1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:55.698 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.698+0000 7f312a3e1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:55.698 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.698+0000 7f312a3e1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:55.698 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.698+0000 7f312a3e1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:55.704 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.703+0000 7f312a3e1700 10 monclient(hunting): handle_auth_done global_id 4180 payload 274 2023-09-30T00:46:55.704 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.703+0000 7f312a3e1700 1 -- 172.21.15.40:0/1456693109 >> v1:172.21.15.135:6789/0 conn(0x7f3124145dc0 legacy=0x7f3124146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:55.704 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.703+0000 7f312a3e1700 1 -- 172.21.15.40:0/1456693109 >> v1:172.21.15.40:6789/0 conn(0x7f312406da60 legacy=0x7f312406dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:55.705 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.703+0000 7f312a3e1700 10 monclient: _finish_hunting 0 2023-09-30T00:46:55.705 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.703+0000 7f312a3e1700 1 monclient: found mon.noname-c 2023-09-30T00:46:55.705 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.703+0000 7f312a3e1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:55.705 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.703+0000 7f312a3e1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:46:55.705 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.703+0000 7f312a3e1700 1 -- 172.21.15.40:0/1456693109 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312406e5f0 con 0x7f31240b1980 2023-09-30T00:46:55.714 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.713+0000 7f312a3e1700 1 --2- 172.21.15.40:0/1456693109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31240b1980 0x7f31240afa20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f311c009e60 tx=0x7f311c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7a4c8c36860dc93 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:55.720 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f31293df700 1 -- 172.21.15.40:0/1456693109 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c00e070 con 0x7f31240b1980 2023-09-30T00:46:55.720 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f31293df700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:55.720 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f31293df700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:55.720 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f31293df700 10 monclient: dump: 2023-09-30T00:46:55.720 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:46:55.720 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:55.721 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:55.721 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:55.721 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:55.721 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:55.721 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:55.721 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:55.721 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:55.721 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:46:55.721 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f31293df700 10 monclient: _finish_auth 0 2023-09-30T00:46:55.721 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f31293df700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:55.722 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f31293df700 1 -- 172.21.15.40:0/1456693109 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311c00bbf0 con 0x7f31240b1980 2023-09-30T00:46:55.722 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f31293df700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:55.722 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f31293df700 1 -- 172.21.15.40:0/1456693109 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311c012430 con 0x7f31240b1980 2023-09-30T00:46:55.722 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f31293df700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:55.722 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f31293df700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:55.722 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f31293df700 10 monclient: dump: 2023-09-30T00:46:55.722 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:46:55.722 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:55.722 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:55.722 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:55.723 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:55.723 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:55.723 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:55.723 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:55.723 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:55.723 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:46:55.723 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f312c645700 5 monclient: authenticate success, global_id 4180 2023-09-30T00:46:55.723 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f312c645700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:55.723 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.719+0000 7f312c645700 4 set_mon_vals no callback set 2023-09-30T00:46:55.725 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.720+0000 7f312c645700 10 monclient: shutdown 2023-09-30T00:46:55.725 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.720+0000 7f312c645700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:55.725 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.720+0000 7f312c645700 1 -- 172.21.15.40:0/1456693109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31240b1980 msgr2=0x7f31240afa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:55.725 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.720+0000 7f312c645700 1 --2- 172.21.15.40:0/1456693109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31240b1980 0x7f31240afa20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f311c009e60 tx=0x7f311c009780 comp rx=0 tx=0).stop 2023-09-30T00:46:55.725 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.720+0000 7f312c645700 1 -- 172.21.15.40:0/1456693109 shutdown_connections 2023-09-30T00:46:55.725 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.720+0000 7f312c645700 1 --2- 172.21.15.40:0/1456693109 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31240b1980 0x7f31240afa20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:55.725 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.720+0000 7f312c645700 1 -- 172.21.15.40:0/1456693109 >> 172.21.15.40:0/1456693109 conn(0x7f3124063280 msgr2=0x7f3124063680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:55.725 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.721+0000 7f312c645700 1 -- 172.21.15.40:0/1456693109 shutdown_connections 2023-09-30T00:46:55.725 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.721+0000 7f312c645700 1 -- 172.21.15.40:0/1456693109 wait complete. 2023-09-30T00:46:55.726 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.721+0000 7f312c645700 10 monclient: build_initial_monmap 2023-09-30T00:46:55.726 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.721+0000 7f312c645700 10 monclient: monmap: 2023-09-30T00:46:55.726 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:46:55.726 INFO:teuthology.misc.health.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:55.726 INFO:teuthology.misc.health.smithi040.stderr:last_changed 0.000000 2023-09-30T00:46:55.726 INFO:teuthology.misc.health.smithi040.stderr:created 0.000000 2023-09-30T00:46:55.726 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:55.726 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2023-09-30T00:46:55.726 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:46:55.726 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:46:55.727 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:46:55.727 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:46:55.727 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.722+0000 7f312c645700 1 librados: starting msgr at 2023-09-30T00:46:55.727 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.722+0000 7f312c645700 1 librados: starting objecter 2023-09-30T00:46:55.727 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.722+0000 7f312c645700 1 Processor -- start 2023-09-30T00:46:55.727 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.722+0000 7f312c645700 1 -- start start 2023-09-30T00:46:55.727 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.722+0000 7f312c645700 1 librados: setting wanted keys 2023-09-30T00:46:55.727 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.722+0000 7f312c645700 1 librados: calling monclient init 2023-09-30T00:46:55.727 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.722+0000 7f312c645700 10 monclient: init 2023-09-30T00:46:55.727 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:55.728 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:46:55.728 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f312406da60 0x7f3124143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:55.728 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 10 monclient(hunting): picked mon.noname-a con 0x7f312406da60 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:46:55.728 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31240b1980 0x7f3124139cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:55.728 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 10 monclient(hunting): picked mon.noname-b con 0x7f31240b1980 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:46:55.728 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124145dc0 0x7f312413a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:55.728 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 10 monclient(hunting): picked mon.noname-c con 0x7f3124145dc0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:46:55.728 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:55.728 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3124145740 con 0x7f312406da60 2023-09-30T00:46:55.728 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:55.729 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3124141be0 con 0x7f31240b1980 2023-09-30T00:46:55.729 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:55.729 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312413a730 con 0x7f3124145dc0 2023-09-30T00:46:55.729 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:55.729 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.723+0000 7f312c645700 10 monclient(hunting): authenticate will time out at 5277.024732s 2023-09-30T00:46:55.729 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.724+0000 7f312abe2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124145dc0 0x7f312413a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:55.730 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.725+0000 7f312abe2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124145dc0 0x7f312413a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:57426/0 (socket says 172.21.15.40:57426) 2023-09-30T00:46:55.730 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.725+0000 7f312abe2700 1 -- 172.21.15.40:0/3647315627 learned_addr learned my addr 172.21.15.40:0/3647315627 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:46:55.730 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.725+0000 7f312abe2700 10 monclient(hunting): get_auth_request con 0x7f3124145dc0 auth_method 0 2023-09-30T00:46:55.730 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.725+0000 7f312abe2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:55.730 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.725+0000 7f312abe2700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:55.730 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.725+0000 7f312abe2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:55.730 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.726+0000 7f312abe2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:55.730 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.726+0000 7f312abe2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:55.730 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.726+0000 7f312abe2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:55.737 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.736+0000 7f312abe2700 10 monclient(hunting): handle_auth_done global_id 4151 payload 824 2023-09-30T00:46:55.737 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.736+0000 7f312abe2700 1 -- 172.21.15.40:0/3647315627 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31240b1980 msgr2=0x7f3124139cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:55.737 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.736+0000 7f312abe2700 1 --2- 172.21.15.40:0/3647315627 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31240b1980 0x7f3124139cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:55.737 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.736+0000 7f312abe2700 1 -- 172.21.15.40:0/3647315627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f312406da60 msgr2=0x7f3124143310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:55.737 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.736+0000 7f312abe2700 1 --2- 172.21.15.40:0/3647315627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f312406da60 0x7f3124143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:55.737 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.736+0000 7f312abe2700 10 monclient: _finish_hunting 0 2023-09-30T00:46:55.737 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.736+0000 7f312abe2700 1 monclient: found mon.noname-c 2023-09-30T00:46:55.737 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.736+0000 7f312abe2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:55.737 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.736+0000 7f312abe2700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:46:55.738 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.736+0000 7f312abe2700 1 -- 172.21.15.40:0/3647315627 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c009a20 con 0x7f3124145dc0 2023-09-30T00:46:55.740 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.739+0000 7f312abe2700 1 --2- 172.21.15.40:0/3647315627 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124145dc0 0x7f312413a1f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f312000b5e0 tx=0x7f3120009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:55.740 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.739+0000 7f311a7fc700 10 client.?.objecter ms_handle_connect 0x7f3124145dc0 2023-09-30T00:46:55.740 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.739+0000 7f311a7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:46:55.747 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f311a7fc700 1 -- 172.21.15.40:0/3647315627 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312000f510 con 0x7f3124145dc0 2023-09-30T00:46:55.748 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f311a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:55.748 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f311a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:55.748 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f311a7fc700 10 monclient: dump: 2023-09-30T00:46:55.748 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:46:55.748 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:55.748 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:55.748 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:55.748 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:55.748 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:55.748 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:55.748 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:55.749 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:55.749 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:46:55.749 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f311a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:46:55.749 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f311a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:55.749 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f311a7fc700 1 -- 172.21.15.40:0/3647315627 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312000f690 con 0x7f3124145dc0 2023-09-30T00:46:55.749 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f311a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:55.749 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f311a7fc700 1 -- 172.21.15.40:0/3647315627 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312000fa50 con 0x7f3124145dc0 2023-09-30T00:46:55.749 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f311a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:55.749 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f311a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:55.749 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f311a7fc700 10 monclient: dump: 2023-09-30T00:46:55.749 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:46:55.750 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:46:55.750 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:55.750 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:46:55.750 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:55.750 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:46:55.750 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:46:55.750 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:46:55.750 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:46:55.750 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:46:55.750 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f312c645700 5 monclient: authenticate success, global_id 4151 2023-09-30T00:46:55.750 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f312c645700 10 monclient: _renew_subs 2023-09-30T00:46:55.751 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f312c645700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:46:55.751 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f312c645700 1 -- 172.21.15.40:0/3647315627 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31241453a0 con 0x7f3124145dc0 2023-09-30T00:46:55.751 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f312c645700 10 client.4151.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:46:55.751 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f312c645700 10 monclient: _renew_subs 2023-09-30T00:46:55.751 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f312c645700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:46:55.751 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f312c645700 1 -- 172.21.15.40:0/3647315627 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3124142950 con 0x7f3124145dc0 2023-09-30T00:46:55.752 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.747+0000 7f312c645700 1 librados: init done 2023-09-30T00:46:55.752 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.748+0000 7f312c645700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:55.752 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.748+0000 7f312c645700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:55.752 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.748+0000 7f312c645700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:46:55.752 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.748+0000 7f312c645700 1 -- 172.21.15.40:0/3647315627 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312406b1a0 con 0x7f3124145dc0 2023-09-30T00:46:55.752 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.749+0000 7f3128bde700 4 set_mon_vals no callback set 2023-09-30T00:46:55.770 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.770+0000 7f311a7fc700 1 -- 172.21.15.40:0/3647315627 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 2) v1 ==== 42252+0+0 (secure 0 0 0) 0x7f3120024430 con 0x7f3124145dc0 2023-09-30T00:46:55.785 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.785+0000 7f311a7fc700 1 -- 172.21.15.40:0/3647315627 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f3120023df0 con 0x7f3124145dc0 2023-09-30T00:46:55.785 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.785+0000 7f311a7fc700 10 client.4151.objecter ms_dispatch 0x7f3124001550 osd_map(10..10 src has 1..10) v4 2023-09-30T00:46:55.786 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.785+0000 7f311a7fc700 3 client.4151.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T00:46:55.786 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.785+0000 7f311a7fc700 3 client.4151.objecter handle_osd_map decoding full epoch 10 2023-09-30T00:46:55.786 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.785+0000 7f311a7fc700 20 client.4151.objecter dump_active .. 0 homeless 2023-09-30T00:46:55.981 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.980+0000 7f311a7fc700 1 -- 172.21.15.40:0/3647315627 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f312000fdc0 con 0x7f3124145dc0 2023-09-30T00:46:55.982 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.980+0000 7f311a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:55.982 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:55.980+0000 7f311a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:56.110 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:56.109+0000 7feac082e700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-30T00:46:56.131 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.130+0000 7f312c645700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:46:56.131 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.130+0000 7f312c645700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:46:56.131 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.130+0000 7f312c645700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:46:56.131 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.130+0000 7f312c645700 1 -- 172.21.15.40:0/3647315627 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3124142950 con 0x7f3124145dc0 2023-09-30T00:46:56.158 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.156+0000 7f311a7fc700 1 -- 172.21.15.40:0/3647315627 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mgrmap(e 3) v1 ==== 42344+0+0 (secure 0 0 0) 0x7f31200246e0 con 0x7f3124145dc0 2023-09-30T00:46:56.158 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.157+0000 7f311a7fc700 1 --2- 172.21.15.40:0/3647315627 >> v2:172.21.15.135:6800/116426 conn(0x7f31080387b0 0x7f310803ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:56.158 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.157+0000 7f312a3e1700 1 --2- 172.21.15.40:0/3647315627 >> v2:172.21.15.135:6800/116426 conn(0x7f31080387b0 0x7f310803ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:56.158 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.157+0000 7f312a3e1700 10 monclient: get_auth_request con 0x7f31080387b0 auth_method 0 2023-09-30T00:46:56.158 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.158+0000 7f311a7fc700 10 client.4151.objecter ms_handle_connect 0x7f31080387b0 2023-09-30T00:46:56.159 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.158+0000 7f312a3e1700 1 --2- 172.21.15.40:0/3647315627 >> v2:172.21.15.135:6800/116426 conn(0x7f31080387b0 0x7f310803ab90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f311c009e60 tx=0x7f311c018000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:56.382 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:46:56.380+0000 7feac082e700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-30T00:46:56.501 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.500+0000 7f311a7fc700 1 -- 172.21.15.40:0/3647315627 <== mon.2 v2:172.21.15.40:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f31200184c0 con 0x7f3124145dc0 2023-09-30T00:46:56.501 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.500+0000 7f311a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:46:56.501 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.500+0000 7f311a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:56.502 INFO:teuthology.misc.health.smithi040.stdout: 2023-09-30T00:46:56.502 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":10,"fsid":"238dba7c-9190-48bd-bb67-322ebc9ecbd0","created":"2023-09-30T00:46:45.822318+0000","modified":"2023-09-30T00:46:52.657791+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:46:52.657791+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"66e1da65-310e-4463-8aa0-431896832e64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d6dd269f-2eda-4ef7-9da1-2ed5b16291b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f0de27d-3674-4ff4-8451-4fcc90786c98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d18ec2f5-7e8f-4732-a523-adc495115cbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e2465fe-bb27-47bd-8ddf-f334cac6041a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7ea449db-9b22-48b8-a61b-1937eeeb6c8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c79b4829-6393-4f9a-a9f8-d2aacb505ae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:46:56.503 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.502+0000 7f312c645700 10 librados: watch_flush enter 2023-09-30T00:46:56.503 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.502+0000 7f312c645700 10 librados: watch_flush exit 2023-09-30T00:46:56.503 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.502+0000 7f312c645700 20 client.4151.objecter shutdown clearing up homeless session... 2023-09-30T00:46:56.503 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.502+0000 7f312c645700 10 client.4151.objecter successfully canceled tick 2023-09-30T00:46:56.503 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.502+0000 7f312c645700 1 -- 172.21.15.40:0/3647315627 >> v2:172.21.15.135:6800/116426 conn(0x7f31080387b0 msgr2=0x7f310803ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:56.503 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.502+0000 7f312c645700 1 --2- 172.21.15.40:0/3647315627 >> v2:172.21.15.135:6800/116426 conn(0x7f31080387b0 0x7f310803ab90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f311c009e60 tx=0x7f311c018000 comp rx=0 tx=0).stop 2023-09-30T00:46:56.503 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.502+0000 7f312c645700 10 monclient: shutdown 2023-09-30T00:46:56.504 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.502+0000 7f312c645700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:56.504 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.502+0000 7f312c645700 1 -- 172.21.15.40:0/3647315627 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124145dc0 msgr2=0x7f312413a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:56.504 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.502+0000 7f312c645700 1 --2- 172.21.15.40:0/3647315627 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124145dc0 0x7f312413a1f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f312000b5e0 tx=0x7f3120009120 comp rx=0 tx=0).stop 2023-09-30T00:46:56.506 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.505+0000 7f312c645700 1 -- 172.21.15.40:0/3647315627 shutdown_connections 2023-09-30T00:46:56.506 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.505+0000 7f312c645700 1 --2- 172.21.15.40:0/3647315627 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3124145dc0 0x7f312413a1f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:56.506 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.505+0000 7f312c645700 1 --2- 172.21.15.40:0/3647315627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f312406da60 0x7f3124143310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:56.506 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.505+0000 7f312c645700 1 --2- 172.21.15.40:0/3647315627 >> v2:172.21.15.135:6800/116426 conn(0x7f31080387b0 0x7f310803ab90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:56.506 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.505+0000 7f312c645700 1 --2- 172.21.15.40:0/3647315627 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31240b1980 0x7f3124139cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:56.506 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.505+0000 7f312c645700 1 -- 172.21.15.40:0/3647315627 >> 172.21.15.40:0/3647315627 conn(0x7f3124063280 msgr2=0x7f31240b51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:56.508 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.507+0000 7f312c645700 1 -- 172.21.15.40:0/3647315627 shutdown_connections 2023-09-30T00:46:56.508 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.507+0000 7f312c645700 1 -- 172.21.15.40:0/3647315627 wait complete. 2023-09-30T00:46:56.508 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:46:56.507+0000 7f312c645700 1 librados: shutdown 2023-09-30T00:46:56.527 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T00:47:03.309 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:10.437 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:03.318 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:10.446 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:03.333 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:47:03.357 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:10.479 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:03.365 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:10.469 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:03.395 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:10.524 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:03.406 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:10.534 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:03.437 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:10.559 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:03.446 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:10.549 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:03.538 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.537+0000 7fd29bea9700 10 monclient: get_monmap_and_config 2023-09-30T00:47:03.538 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.537+0000 7fd29bea9700 10 monclient: build_initial_monmap 2023-09-30T00:47:03.538 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.537+0000 7fd29bea9700 10 monclient: monmap: 2023-09-30T00:47:03.538 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:47:03.538 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:03.538 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:47:03.538276+0000 2023-09-30T00:47:03.538 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:47:03.538276+0000 2023-09-30T00:47:03.538 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:03.539 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:03.539 INFO:teuthology.misc.health.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:03.539 INFO:teuthology.misc.health.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:03.539 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:03.539 INFO:teuthology.misc.health.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:03.539 INFO:teuthology.misc.health.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:03.539 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:03.540 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.538+0000 7fd29bea9700 1 Processor -- start 2023-09-30T00:47:03.540 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.538+0000 7fd29bea9700 1 -- start start 2023-09-30T00:47:03.540 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.538+0000 7fd29bea9700 10 monclient: init 2023-09-30T00:47:03.540 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.539+0000 7fd29bea9700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:03.540 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.539+0000 7fd29bea9700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-09-30T00:47:03.540 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.539+0000 7fd29bea9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2940a5be0 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:03.540 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.539+0000 7fd29bea9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2940a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:03.540 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.539+0000 7fd29bea9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd2940a63e0 0x7fd294142410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:03.540 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.539+0000 7fd29bea9700 10 monclient(hunting): picked mon.noname-a con 0x7fd2940a63e0 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:03.540 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.539+0000 7fd29bea9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2940a7750 con 0x7fd2940a5be0 2023-09-30T00:47:03.540 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.539+0000 7fd29bea9700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2940a59d0 con 0x7fd2940a5fb0 2023-09-30T00:47:03.541 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.539+0000 7fd29bea9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:03.541 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.539+0000 7fd29bea9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd294142950 con 0x7fd2940a63e0 2023-09-30T00:47:03.541 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.539+0000 7fd29bea9700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:03.541 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.539+0000 7fd29bea9700 10 monclient(hunting): authenticate will time out at 5284.840529s 2023-09-30T00:47:03.542 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.540+0000 7fd299444700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fd2940a5fb0 0x7fd294141d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:33608/0 (socket says 172.21.15.40:33608) 2023-09-30T00:47:03.542 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.540+0000 7fd299444700 1 -- 172.21.15.40:0/3305015090 learned_addr learned my addr 172.21.15.40:0/3305015090 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:03.542 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.540+0000 7fd29a446700 1 --2- 172.21.15.40:0/3305015090 >> v2:172.21.15.40:3300/0 conn(0x7fd2940a63e0 0x7fd294142410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:03.542 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.541+0000 7fd29a446700 10 monclient(hunting): get_auth_request con 0x7fd2940a63e0 auth_method 0 2023-09-30T00:47:03.542 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.541+0000 7fd29a446700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:03.542 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.541+0000 7fd29a446700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:03.542 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.541+0000 7fd29a446700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:03.542 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.542+0000 7fd29a446700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:03.543 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.542+0000 7fd29a446700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:03.543 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.542+0000 7fd29a446700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:03.543 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.543+0000 7fd298c43700 1 -- 172.21.15.40:0/3305015090 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64629125 0 0) 0x7fd2940a59d0 con 0x7fd2940a5fb0 2023-09-30T00:47:03.543 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.543+0000 7fd298c43700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:03.543 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.543+0000 7fd298c43700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:03.544 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.543+0000 7fd298c43700 10 monclient(hunting): my global_id is 4201 2023-09-30T00:47:03.544 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.543+0000 7fd298c43700 1 -- 172.21.15.40:0/3305015090 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd280003570 con 0x7fd2940a5fb0 2023-09-30T00:47:03.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.545+0000 7fd29a446700 10 monclient(hunting): handle_auth_done global_id 4176 payload 274 2023-09-30T00:47:03.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.545+0000 7fd29a446700 1 -- 172.21.15.40:0/3305015090 >> v1:172.21.15.135:6789/0 conn(0x7fd2940a5fb0 legacy=0x7fd294141d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:03.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.545+0000 7fd29a446700 1 -- 172.21.15.40:0/3305015090 >> v1:172.21.15.40:6789/0 conn(0x7fd2940a5be0 legacy=0x7fd29413eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:03.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.545+0000 7fd29a446700 10 monclient: _finish_hunting 0 2023-09-30T00:47:03.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.545+0000 7fd29a446700 1 monclient: found mon.noname-a 2023-09-30T00:47:03.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.545+0000 7fd29a446700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:03.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.545+0000 7fd29a446700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:03.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.545+0000 7fd29a446700 1 -- 172.21.15.40:0/3305015090 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2940a56f0 con 0x7fd2940a63e0 2023-09-30T00:47:03.547 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.547+0000 7fd29a446700 1 --2- 172.21.15.40:0/3305015090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2940a63e0 0x7fd294142410 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd28c009e60 tx=0x7fd28c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e0c6449a700af8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:03.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.552+0000 7fd298c43700 1 -- 172.21.15.40:0/3305015090 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c00e070 con 0x7fd2940a63e0 2023-09-30T00:47:03.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd298c43700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:03.554 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd298c43700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:03.554 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd298c43700 10 monclient: dump: 2023-09-30T00:47:03.554 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:03.554 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:03.554 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:03.554 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:03.554 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:03.554 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:03.554 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:03.554 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:03.555 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:03.555 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:03.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd298c43700 10 monclient: _finish_auth 0 2023-09-30T00:47:03.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd298c43700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:03.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd298c43700 1 -- 172.21.15.40:0/3305015090 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c00bb20 con 0x7fd2940a63e0 2023-09-30T00:47:03.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd29bea9700 5 monclient: authenticate success, global_id 4176 2023-09-30T00:47:03.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd29bea9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:03.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd298c43700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:03.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd298c43700 1 -- 172.21.15.40:0/3305015090 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c012430 con 0x7fd2940a63e0 2023-09-30T00:47:03.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd298c43700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:03.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd298c43700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:03.556 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd298c43700 10 monclient: dump: 2023-09-30T00:47:03.556 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:03.556 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:03.556 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:03.556 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:03.556 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:03.556 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:03.556 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:03.556 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:03.556 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:03.556 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:03.557 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd29bea9700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:03.557 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd29bea9700 4 set_mon_vals no callback set 2023-09-30T00:47:03.562 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd29bea9700 10 monclient: shutdown 2023-09-30T00:47:03.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd29bea9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:03.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd29bea9700 1 -- 172.21.15.40:0/3305015090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2940a63e0 msgr2=0x7fd294142410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:03.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.553+0000 7fd29bea9700 1 --2- 172.21.15.40:0/3305015090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2940a63e0 0x7fd294142410 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd28c009e60 tx=0x7fd28c009780 comp rx=0 tx=0).stop 2023-09-30T00:47:03.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.554+0000 7fd29bea9700 1 -- 172.21.15.40:0/3305015090 shutdown_connections 2023-09-30T00:47:03.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.554+0000 7fd29bea9700 1 --2- 172.21.15.40:0/3305015090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2940a63e0 0x7fd294142410 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:03.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.554+0000 7fd29bea9700 1 -- 172.21.15.40:0/3305015090 >> 172.21.15.40:0/3305015090 conn(0x7fd294063280 msgr2=0x7fd294063680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:03.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.554+0000 7fd29bea9700 1 -- 172.21.15.40:0/3305015090 shutdown_connections 2023-09-30T00:47:03.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.554+0000 7fd29bea9700 1 -- 172.21.15.40:0/3305015090 wait complete. 2023-09-30T00:47:03.564 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.555+0000 7fd29bea9700 10 monclient: build_initial_monmap 2023-09-30T00:47:03.564 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.555+0000 7fd29bea9700 10 monclient: monmap: 2023-09-30T00:47:03.564 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:47:03.564 INFO:teuthology.misc.health.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:03.564 INFO:teuthology.misc.health.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:03.564 INFO:teuthology.misc.health.smithi040.stderr:created 0.000000 2023-09-30T00:47:03.564 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:03.564 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:03.564 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:03.564 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:03.565 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:03.565 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:03.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.555+0000 7fd29bea9700 1 librados: starting msgr at 2023-09-30T00:47:03.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.555+0000 7fd29bea9700 1 librados: starting objecter 2023-09-30T00:47:03.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.555+0000 7fd29bea9700 1 Processor -- start 2023-09-30T00:47:03.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.555+0000 7fd29bea9700 1 -- start start 2023-09-30T00:47:03.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.555+0000 7fd29bea9700 1 librados: setting wanted keys 2023-09-30T00:47:03.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.555+0000 7fd29bea9700 1 librados: calling monclient init 2023-09-30T00:47:03.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.555+0000 7fd29bea9700 10 monclient: init 2023-09-30T00:47:03.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:03.566 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:47:03.566 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2940a5be0 0x7fd2940aa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:03.566 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 10 monclient(hunting): picked mon.noname-a con 0x7fd2940a5be0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:03.566 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2940a5fb0 0x7fd2940a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:03.566 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 10 monclient(hunting): picked mon.noname-b con 0x7fd2940a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:03.566 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2940a63e0 0x7fd2940a8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:03.566 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 10 monclient(hunting): picked mon.noname-c con 0x7fd2940a63e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:03.566 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:03.566 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2940ab370 con 0x7fd2940a5be0 2023-09-30T00:47:03.566 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:03.567 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2940aaae0 con 0x7fd2940a5fb0 2023-09-30T00:47:03.567 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:03.567 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2940a9980 con 0x7fd2940a63e0 2023-09-30T00:47:03.567 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:03.567 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.556+0000 7fd29bea9700 10 monclient(hunting): authenticate will time out at 5284.857822s 2023-09-30T00:47:03.567 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.557+0000 7fd299c45700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2940a5be0 0x7fd2940aa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:03.567 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.557+0000 7fd299444700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2940a5fb0 0x7fd2940a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:03.567 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.557+0000 7fd299c45700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2940a5be0 0x7fd2940aa5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48706/0 (socket says 172.21.15.40:48706) 2023-09-30T00:47:03.567 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.557+0000 7fd299c45700 1 -- 172.21.15.40:0/1443282482 learned_addr learned my addr 172.21.15.40:0/1443282482 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:03.567 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.557+0000 7fd299c45700 10 monclient(hunting): get_auth_request con 0x7fd2940a5be0 auth_method 0 2023-09-30T00:47:03.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.557+0000 7fd299c45700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:03.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.557+0000 7fd299c45700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:03.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.557+0000 7fd299c45700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:03.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.557+0000 7fd299444700 10 monclient(hunting): get_auth_request con 0x7fd2940a5fb0 auth_method 0 2023-09-30T00:47:03.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.557+0000 7fd299444700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:03.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.557+0000 7fd299444700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:03.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.557+0000 7fd299444700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:03.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.558+0000 7fd29a446700 1 --2- 172.21.15.40:0/1443282482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2940a63e0 0x7fd2940a8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:03.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.558+0000 7fd299444700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:03.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.558+0000 7fd299444700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:03.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.558+0000 7fd299444700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:03.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.558+0000 7fd299c45700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:03.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.558+0000 7fd299c45700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:03.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.558+0000 7fd299c45700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:03.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.558+0000 7fd29a446700 10 monclient(hunting): get_auth_request con 0x7fd2940a63e0 auth_method 0 2023-09-30T00:47:03.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.558+0000 7fd29a446700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:03.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.558+0000 7fd29a446700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:03.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.558+0000 7fd29a446700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:03.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.559+0000 7fd29a446700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:03.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.559+0000 7fd29a446700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:03.570 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.559+0000 7fd29a446700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:03.570 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.561+0000 7fd299444700 10 monclient(hunting): handle_auth_done global_id 4204 payload 824 2023-09-30T00:47:03.570 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.561+0000 7fd299444700 1 -- 172.21.15.40:0/1443282482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2940a63e0 msgr2=0x7fd2940a8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:03.570 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.561+0000 7fd299444700 1 --2- 172.21.15.40:0/1443282482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2940a63e0 0x7fd2940a8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:03.570 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.561+0000 7fd299444700 1 -- 172.21.15.40:0/1443282482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2940a5be0 msgr2=0x7fd2940aa5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:03.570 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.561+0000 7fd299444700 1 --2- 172.21.15.40:0/1443282482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2940a5be0 0x7fd2940aa5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:03.570 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.561+0000 7fd299444700 10 monclient: _finish_hunting 0 2023-09-30T00:47:03.570 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.561+0000 7fd299444700 1 monclient: found mon.noname-b 2023-09-30T00:47:03.570 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.561+0000 7fd299444700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:03.571 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.561+0000 7fd299444700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:47:03.571 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.561+0000 7fd299444700 1 -- 172.21.15.40:0/1443282482 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c009a20 con 0x7fd2940a5fb0 2023-09-30T00:47:03.572 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.563+0000 7fd299444700 1 --2- 172.21.15.40:0/1443282482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2940a5fb0 0x7fd2940a7c00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd28400c580 tx=0x7fd284007d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:03.572 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.563+0000 7fd289ffb700 10 client.?.objecter ms_handle_connect 0x7fd2940a5fb0 2023-09-30T00:47:03.572 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.563+0000 7fd289ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:03.572 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.567+0000 7fd289ffb700 1 -- 172.21.15.40:0/1443282482 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd284005b30 con 0x7fd2940a5fb0 2023-09-30T00:47:03.572 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.567+0000 7fd289ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:03.572 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.567+0000 7fd289ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:03.572 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.567+0000 7fd289ffb700 10 monclient: dump: 2023-09-30T00:47:03.572 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:03.573 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:03.573 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:03.573 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:03.573 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:03.573 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:03.573 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:03.573 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:03.573 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:03.573 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:03.573 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.567+0000 7fd289ffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:03.573 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.567+0000 7fd289ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:03.574 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.567+0000 7fd289ffb700 1 -- 172.21.15.40:0/1443282482 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd284005c90 con 0x7fd2940a5fb0 2023-09-30T00:47:03.574 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.567+0000 7fd289ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:03.574 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.567+0000 7fd289ffb700 1 -- 172.21.15.40:0/1443282482 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd284007320 con 0x7fd2940a5fb0 2023-09-30T00:47:03.574 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.567+0000 7fd289ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:03.574 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.567+0000 7fd289ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:03.574 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.567+0000 7fd289ffb700 10 monclient: dump: 2023-09-30T00:47:03.574 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:03.574 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:03.574 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:03.575 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:03.575 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:03.575 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:03.575 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:03.575 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:03.575 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:03.575 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:03.575 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.567+0000 7fd28bfff700 4 set_mon_vals no callback set 2023-09-30T00:47:03.575 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.568+0000 7fd29bea9700 5 monclient: authenticate success, global_id 4204 2023-09-30T00:47:03.575 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.568+0000 7fd29bea9700 10 monclient: _renew_subs 2023-09-30T00:47:03.575 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.568+0000 7fd29bea9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:03.576 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.568+0000 7fd29bea9700 1 -- 172.21.15.40:0/1443282482 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29413b0e0 con 0x7fd2940a5fb0 2023-09-30T00:47:03.576 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.568+0000 7fd29bea9700 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:03.576 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.568+0000 7fd29bea9700 10 monclient: _renew_subs 2023-09-30T00:47:03.576 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.568+0000 7fd29bea9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:03.576 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.568+0000 7fd29bea9700 1 -- 172.21.15.40:0/1443282482 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2940aba50 con 0x7fd2940a5fb0 2023-09-30T00:47:03.576 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.568+0000 7fd29bea9700 1 librados: init done 2023-09-30T00:47:03.576 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.569+0000 7fd29bea9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:03.576 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.569+0000 7fd29bea9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:03.576 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.569+0000 7fd29bea9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:03.576 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.569+0000 7fd29bea9700 1 -- 172.21.15.40:0/1443282482 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2940acd50 con 0x7fd2940a5fb0 2023-09-30T00:47:03.794 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.793+0000 7fd289ffb700 1 -- 172.21.15.40:0/1443282482 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7fd28401b3b0 con 0x7fd2940a5fb0 2023-09-30T00:47:03.795 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.794+0000 7fd289ffb700 1 --2- 172.21.15.40:0/1443282482 >> v2:172.21.15.135:6800/116426 conn(0x7fd278068800 0x7fd27806acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:03.795 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.794+0000 7fd299c45700 1 --2- 172.21.15.40:0/1443282482 >> v2:172.21.15.135:6800/116426 conn(0x7fd278068800 0x7fd27806acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:03.795 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.794+0000 7fd299c45700 10 monclient: get_auth_request con 0x7fd278068800 auth_method 0 2023-09-30T00:47:03.796 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.795+0000 7fd289ffb700 10 client.4204.objecter ms_handle_connect 0x7fd278068800 2023-09-30T00:47:03.796 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.795+0000 7fd299c45700 1 --2- 172.21.15.40:0/1443282482 >> v2:172.21.15.135:6800/116426 conn(0x7fd278068800 0x7fd27806acb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd290009b80 tx=0x7fd2900091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:03.817 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.816+0000 7fd289ffb700 1 -- 172.21.15.40:0/1443282482 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fd284055410 con 0x7fd2940a5fb0 2023-09-30T00:47:03.817 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.816+0000 7fd289ffb700 10 client.4204.objecter ms_dispatch 0x7fd294001550 osd_map(10..10 src has 1..10) v4 2023-09-30T00:47:03.817 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.816+0000 7fd289ffb700 3 client.4204.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T00:47:03.817 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.816+0000 7fd289ffb700 3 client.4204.objecter handle_osd_map decoding full epoch 10 2023-09-30T00:47:03.817 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.816+0000 7fd289ffb700 20 client.4204.objecter dump_active .. 0 homeless 2023-09-30T00:47:03.833 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.833+0000 7fd289ffb700 1 -- 172.21.15.40:0/1443282482 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd284051050 con 0x7fd2940a5fb0 2023-09-30T00:47:03.834 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.833+0000 7fd289ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:03.834 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:03.833+0000 7fd289ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:03.932 INFO:tasks.ceph.osd.5.smithi135.stderr:2023-09-30T00:47:03.923+0000 40451c0 -1 Falling back to public interface 2023-09-30T00:47:03.963 INFO:tasks.ceph.osd.4.smithi135.stderr:2023-09-30T00:47:03.950+0000 40451c0 -1 Falling back to public interface 2023-09-30T00:47:03.998 INFO:tasks.ceph.osd.7.smithi135.stderr:2023-09-30T00:47:03.990+0000 40451c0 -1 Falling back to public interface 2023-09-30T00:47:04.035 INFO:tasks.ceph.osd.6.smithi135.stderr:2023-09-30T00:47:04.023+0000 40451c0 -1 Falling back to public interface 2023-09-30T00:47:04.091 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.090+0000 7fd29bea9700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:04.091 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.090+0000 7fd29bea9700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:04.091 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.090+0000 7fd29bea9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:04.091 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.090+0000 7fd29bea9700 1 -- 172.21.15.40:0/1443282482 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd294064a00 con 0x7fd2940a5fb0 2023-09-30T00:47:04.184 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.183+0000 7fd289ffb700 1 -- 172.21.15.40:0/1443282482 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fd284017020 con 0x7fd2940a5fb0 2023-09-30T00:47:04.184 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.183+0000 7fd289ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:04.184 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.183+0000 7fd289ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:04.185 INFO:teuthology.misc.health.smithi040.stdout: 2023-09-30T00:47:04.185 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":10,"fsid":"238dba7c-9190-48bd-bb67-322ebc9ecbd0","created":"2023-09-30T00:46:45.822318+0000","modified":"2023-09-30T00:46:52.657791+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:46:52.657791+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"66e1da65-310e-4463-8aa0-431896832e64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d6dd269f-2eda-4ef7-9da1-2ed5b16291b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f0de27d-3674-4ff4-8451-4fcc90786c98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d18ec2f5-7e8f-4732-a523-adc495115cbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e2465fe-bb27-47bd-8ddf-f334cac6041a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7ea449db-9b22-48b8-a61b-1937eeeb6c8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c79b4829-6393-4f9a-a9f8-d2aacb505ae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:47:04.186 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.186+0000 7fd29bea9700 10 librados: watch_flush enter 2023-09-30T00:47:04.187 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.186+0000 7fd29bea9700 10 librados: watch_flush exit 2023-09-30T00:47:04.187 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.186+0000 7fd29bea9700 20 client.4204.objecter shutdown clearing up homeless session... 2023-09-30T00:47:04.187 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.186+0000 7fd29bea9700 10 client.4204.objecter successfully canceled tick 2023-09-30T00:47:04.187 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.186+0000 7fd29bea9700 1 -- 172.21.15.40:0/1443282482 >> v2:172.21.15.135:6800/116426 conn(0x7fd278068800 msgr2=0x7fd27806acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:04.187 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.186+0000 7fd29bea9700 1 --2- 172.21.15.40:0/1443282482 >> v2:172.21.15.135:6800/116426 conn(0x7fd278068800 0x7fd27806acb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd290009b80 tx=0x7fd2900091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:04.187 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.186+0000 7fd29bea9700 10 monclient: shutdown 2023-09-30T00:47:04.187 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.186+0000 7fd29bea9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:04.187 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.186+0000 7fd29bea9700 1 -- 172.21.15.40:0/1443282482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2940a5fb0 msgr2=0x7fd2940a7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:04.187 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.186+0000 7fd29bea9700 1 --2- 172.21.15.40:0/1443282482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2940a5fb0 0x7fd2940a7c00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd28400c580 tx=0x7fd284007d50 comp rx=0 tx=0).stop 2023-09-30T00:47:04.188 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.187+0000 7fd29bea9700 1 -- 172.21.15.40:0/1443282482 shutdown_connections 2023-09-30T00:47:04.188 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.187+0000 7fd29bea9700 1 --2- 172.21.15.40:0/1443282482 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd2940a63e0 0x7fd2940a8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:04.188 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.187+0000 7fd29bea9700 1 --2- 172.21.15.40:0/1443282482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd2940a5be0 0x7fd2940aa5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:04.188 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.187+0000 7fd29bea9700 1 --2- 172.21.15.40:0/1443282482 >> v2:172.21.15.135:6800/116426 conn(0x7fd278068800 0x7fd27806acb0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:04.188 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.187+0000 7fd29bea9700 1 --2- 172.21.15.40:0/1443282482 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd2940a5fb0 0x7fd2940a7c00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:04.188 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.187+0000 7fd29bea9700 1 -- 172.21.15.40:0/1443282482 >> 172.21.15.40:0/1443282482 conn(0x7fd294063280 msgr2=0x7fd294063650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:04.189 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.189+0000 7fd29bea9700 1 -- 172.21.15.40:0/1443282482 shutdown_connections 2023-09-30T00:47:04.189 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.189+0000 7fd29bea9700 1 -- 172.21.15.40:0/1443282482 wait complete. 2023-09-30T00:47:04.189 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:04.189+0000 7fd29bea9700 1 librados: shutdown 2023-09-30T00:47:04.205 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T00:47:04.593 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:11.721 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:04.639 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:11.761 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:04.667 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:11.795 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:04.702 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:11.805 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:04.780 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:11.888 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:04.787 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:11.915 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:04.879 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:11.986 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:04.970 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:12.098 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.017 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:12.139 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.063 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:12.164 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.074 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:12.177 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.128 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:12.228 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.139 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:12.267 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.151 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:12.265 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.160 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:12.262 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.220 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:12.320 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.240 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:12.354 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.341 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:12.469 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.389 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:12.511 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.413 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-09-30T00:47:05.405+0000 40451c0 -1 Falling back to public interface 2023-09-30T00:47:05.435 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:12.563 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.457 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:12.561 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.637 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:12.765 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.683 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:12.805 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.710 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-09-30T00:47:05.702+0000 40451c0 -1 Falling back to public interface 2023-09-30T00:47:05.733 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:12.861 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.757 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:12.860 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.768 INFO:tasks.ceph.osd.2.smithi040.stderr:2023-09-30T00:47:05.758+0000 40451c0 -1 Falling back to public interface 2023-09-30T00:47:05.799 INFO:tasks.ceph.osd.3.smithi040.stderr:2023-09-30T00:47:05.790+0000 40451c0 -1 Falling back to public interface 2023-09-30T00:47:05.808 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:12.911 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.928 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:13.057 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:05.974 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:13.096 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.037 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:13.165 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.105 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:13.209 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.123 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:13.231 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.221 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:13.349 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.269 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:13.391 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.301 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:13.429 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.369 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:13.473 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.400 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:13.501 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.445 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:13.546 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.484 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:13.598 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.488 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:13.616 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.499 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:13.606 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.534 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:13.655 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.565 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:13.694 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.649 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:13.752 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.754 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:13.883 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.779 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:13.881 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.798 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:13.920 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.825 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:13.926 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.829 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:13.958 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.831 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:13.939 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.864 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:13.978 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:06.914 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:14.017 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.429 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:14.205 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.430 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:14.147 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.430 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:14.216 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.431 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:14.185 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.431 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:14.257 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.434 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:14.305 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.434 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:14.325 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.434 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:14.394 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.434 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:14.478 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.435 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:14.489 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.435 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:14.382 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.435 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:14.451 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.435 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:14.523 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.436 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:14.512 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.436 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:14.552 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.454 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:14.582 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.457 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:14.571 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.501 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:14.623 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.864 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:14.743 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.864 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:14.760 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.864 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:14.792 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.865 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:14.839 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.865 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:14.865 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.900 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:15.008 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.928 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:15.030 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:07.960 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:15.061 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:08.039 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:15.153 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:08.186 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:15.294 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:08.214 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:15.315 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:08.253 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:15.354 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:08.333 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:15.446 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:08.481 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:15.588 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:08.517 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:15.618 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:08.553 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:15.653 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:08.631 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:15.745 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:08.814 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:15.915 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:08.852 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:15.952 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:08.915 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:16.022 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:09.011 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:16.119 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:09.087 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:16.201 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:09.167 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:16.281 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:09.225 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:16.326 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:09.304 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:16.405 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:09.315 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:16.416 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:09.389 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:16.489 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:11.012 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:47:11.172 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.171+0000 7ff9f9282700 10 monclient: get_monmap_and_config 2023-09-30T00:47:11.172 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.171+0000 7ff9f9282700 10 monclient: build_initial_monmap 2023-09-30T00:47:11.172 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.171+0000 7ff9f9282700 10 monclient: monmap: 2023-09-30T00:47:11.173 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:47:11.173 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:11.173 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:47:11.172929+0000 2023-09-30T00:47:11.173 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:47:11.172929+0000 2023-09-30T00:47:11.173 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:11.173 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:11.173 INFO:teuthology.misc.health.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:11.173 INFO:teuthology.misc.health.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:11.173 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:11.173 INFO:teuthology.misc.health.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:11.173 INFO:teuthology.misc.health.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:11.173 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:11.174 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.172+0000 7ff9f9282700 1 Processor -- start 2023-09-30T00:47:11.174 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.172+0000 7ff9f9282700 1 -- start start 2023-09-30T00:47:11.174 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.172+0000 7ff9f9282700 10 monclient: init 2023-09-30T00:47:11.174 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.173+0000 7ff9f9282700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:11.175 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.173+0000 7ff9f9282700 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-09-30T00:47:11.175 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.173+0000 7ff9f9282700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9f40a5be0 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:11.175 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.173+0000 7ff9f9282700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ff9f40a5fb0 0x7ff9f4141d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:11.175 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.173+0000 7ff9f9282700 10 monclient(hunting): picked mon.noname-c con 0x7ff9f40a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:11.175 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.173+0000 7ff9f9282700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9f40a63e0 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:11.175 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.173+0000 7ff9f9282700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f40a7750 con 0x7ff9f40a63e0 2023-09-30T00:47:11.175 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.173+0000 7ff9f9282700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f40a59d0 con 0x7ff9f40a5be0 2023-09-30T00:47:11.175 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.173+0000 7ff9f9282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:11.175 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.173+0000 7ff9f9282700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f4142950 con 0x7ff9f40a5fb0 2023-09-30T00:47:11.175 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.173+0000 7ff9f9282700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:11.175 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.173+0000 7ff9f9282700 10 monclient(hunting): authenticate will time out at 5292.474555s 2023-09-30T00:47:11.176 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.174+0000 7ff9f37fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff9f40a63e0 0x7ff9f4142240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38562/0 (socket says 172.21.15.40:38562) 2023-09-30T00:47:11.176 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.174+0000 7ff9f37fe700 1 -- 172.21.15.40:0/2994475159 learned_addr learned my addr 172.21.15.40:0/2994475159 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:11.179 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.178+0000 7ff9f27fc700 1 --2- 172.21.15.40:0/2994475159 >> v2:172.21.15.135:3300/0 conn(0x7ff9f40a5fb0 0x7ff9f4141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:11.180 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.179+0000 7ff9f27fc700 10 monclient(hunting): get_auth_request con 0x7ff9f40a5fb0 auth_method 0 2023-09-30T00:47:11.180 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.179+0000 7ff9f27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:11.180 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.179+0000 7ff9f27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:11.180 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.179+0000 7ff9f27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:11.181 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.180+0000 7ff9f27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:11.181 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.180+0000 7ff9f27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:11.181 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.180+0000 7ff9f27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:11.182 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.182+0000 7ff9f1ffb700 1 -- 172.21.15.40:0/2994475159 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160205855 0 0) 0x7ff9f40a7750 con 0x7ff9f40a63e0 2023-09-30T00:47:11.182 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.182+0000 7ff9f1ffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:11.182 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.182+0000 7ff9f1ffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:11.183 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.182+0000 7ff9f1ffb700 10 monclient(hunting): my global_id is 4197 2023-09-30T00:47:11.183 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.182+0000 7ff9f1ffb700 1 -- 172.21.15.40:0/2994475159 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9dc003570 con 0x7ff9f40a63e0 2023-09-30T00:47:11.189 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.189+0000 7ff9f27fc700 10 monclient(hunting): handle_auth_done global_id 4222 payload 274 2023-09-30T00:47:11.189 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.189+0000 7ff9f27fc700 1 -- 172.21.15.40:0/2994475159 >> v1:172.21.15.135:6789/0 conn(0x7ff9f40a5be0 legacy=0x7ff9f413eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:11.189 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.189+0000 7ff9f27fc700 1 -- 172.21.15.40:0/2994475159 >> v1:172.21.15.40:6789/0 conn(0x7ff9f40a63e0 legacy=0x7ff9f4142240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:11.189 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.189+0000 7ff9f27fc700 10 monclient: _finish_hunting 0 2023-09-30T00:47:11.190 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.189+0000 7ff9f27fc700 1 monclient: found mon.noname-c 2023-09-30T00:47:11.190 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.189+0000 7ff9f27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:11.190 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.189+0000 7ff9f27fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:47:11.190 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.189+0000 7ff9f27fc700 1 -- 172.21.15.40:0/2994475159 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f40a56f0 con 0x7ff9f40a5fb0 2023-09-30T00:47:11.194 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.194+0000 7ff9f27fc700 1 --2- 172.21.15.40:0/2994475159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9f40a5fb0 0x7ff9f4141d00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80098e0 tx=0x7ff9e80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1da8993dd92765ff server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:11.202 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.201+0000 7ff9f1ffb700 1 -- 172.21.15.40:0/2994475159 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e800b640 con 0x7ff9f40a5fb0 2023-09-30T00:47:11.202 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.201+0000 7ff9f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:11.202 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.201+0000 7ff9f1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:11.203 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.201+0000 7ff9f1ffb700 10 monclient: dump: 2023-09-30T00:47:11.203 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:11.203 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:11.203 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:11.203 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:11.203 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:11.203 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:11.203 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:11.203 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:11.203 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:11.203 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:11.203 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.201+0000 7ff9f1ffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:11.204 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.201+0000 7ff9f1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:11.204 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f1ffb700 1 -- 172.21.15.40:0/2994475159 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e800b7a0 con 0x7ff9f40a5fb0 2023-09-30T00:47:11.204 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:11.204 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f1ffb700 1 -- 172.21.15.40:0/2994475159 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e800bb40 con 0x7ff9f40a5fb0 2023-09-30T00:47:11.204 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:11.204 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:11.204 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f1ffb700 10 monclient: dump: 2023-09-30T00:47:11.204 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:11.204 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:11.204 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:11.204 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:11.204 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:11.205 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:11.205 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:11.205 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:11.205 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:11.205 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:11.205 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f9282700 5 monclient: authenticate success, global_id 4222 2023-09-30T00:47:11.205 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f9282700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:11.205 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f9282700 4 set_mon_vals no callback set 2023-09-30T00:47:11.207 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f9282700 10 monclient: shutdown 2023-09-30T00:47:11.207 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f9282700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:11.207 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f9282700 1 -- 172.21.15.40:0/2994475159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9f40a5fb0 msgr2=0x7ff9f4141d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:11.207 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f9282700 1 --2- 172.21.15.40:0/2994475159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9f40a5fb0 0x7ff9f4141d00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80098e0 tx=0x7ff9e80091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:11.207 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f9282700 1 -- 172.21.15.40:0/2994475159 shutdown_connections 2023-09-30T00:47:11.207 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f9282700 1 --2- 172.21.15.40:0/2994475159 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9f40a5fb0 0x7ff9f4141d00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:11.207 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f9282700 1 -- 172.21.15.40:0/2994475159 >> 172.21.15.40:0/2994475159 conn(0x7ff9f4063280 msgr2=0x7ff9f4063680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:11.207 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f9282700 1 -- 172.21.15.40:0/2994475159 shutdown_connections 2023-09-30T00:47:11.207 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.202+0000 7ff9f9282700 1 -- 172.21.15.40:0/2994475159 wait complete. 2023-09-30T00:47:11.207 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.203+0000 7ff9f9282700 10 monclient: build_initial_monmap 2023-09-30T00:47:11.207 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.203+0000 7ff9f9282700 10 monclient: monmap: 2023-09-30T00:47:11.208 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:47:11.208 INFO:teuthology.misc.health.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:11.208 INFO:teuthology.misc.health.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:11.208 INFO:teuthology.misc.health.smithi040.stderr:created 0.000000 2023-09-30T00:47:11.208 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:11.208 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:11.208 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:11.208 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:11.208 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:11.208 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:11.208 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 1 librados: starting msgr at 2023-09-30T00:47:11.208 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 1 librados: starting objecter 2023-09-30T00:47:11.209 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 1 Processor -- start 2023-09-30T00:47:11.209 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 1 -- start start 2023-09-30T00:47:11.209 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 1 librados: setting wanted keys 2023-09-30T00:47:11.209 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 1 librados: calling monclient init 2023-09-30T00:47:11.209 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 10 monclient: init 2023-09-30T00:47:11.209 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:11.209 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:47:11.209 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9f40a5be0 0x7ff9f40aaba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:11.209 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 10 monclient(hunting): picked mon.noname-a con 0x7ff9f40a5be0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:11.209 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9f40a5fb0 0x7ff9f413d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:11.209 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 10 monclient(hunting): picked mon.noname-b con 0x7ff9f40a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:11.210 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9f40a63e0 0x7ff9f4151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:11.211 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 10 monclient(hunting): picked mon.noname-c con 0x7ff9f40a63e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:11.211 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:11.211 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f40aa710 con 0x7ff9f40a5be0 2023-09-30T00:47:11.211 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:11.211 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f40abd30 con 0x7ff9f40a5fb0 2023-09-30T00:47:11.211 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:11.211 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f40ac010 con 0x7ff9f40a63e0 2023-09-30T00:47:11.211 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:11.211 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.204+0000 7ff9f9282700 10 monclient(hunting): authenticate will time out at 5292.505840s 2023-09-30T00:47:11.211 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.205+0000 7ff9f27fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9f40a5fb0 0x7ff9f413d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:11.211 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.205+0000 7ff9f37fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9f40a63e0 0x7ff9f4151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:11.212 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.205+0000 7ff9f27fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9f40a5fb0 0x7ff9f413d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:59912/0 (socket says 172.21.15.40:59912) 2023-09-30T00:47:11.212 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.205+0000 7ff9f27fc700 1 -- 172.21.15.40:0/2168013181 learned_addr learned my addr 172.21.15.40:0/2168013181 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:11.212 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.205+0000 7ff9f27fc700 10 monclient(hunting): get_auth_request con 0x7ff9f40a5fb0 auth_method 0 2023-09-30T00:47:11.212 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.205+0000 7ff9f27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:11.212 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.205+0000 7ff9f27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:11.212 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.205+0000 7ff9f27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:11.212 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.206+0000 7ff9f37fe700 10 monclient(hunting): get_auth_request con 0x7ff9f40a63e0 auth_method 0 2023-09-30T00:47:11.212 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.206+0000 7ff9f37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:11.212 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.206+0000 7ff9f37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:11.212 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.206+0000 7ff9f37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:11.212 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.206+0000 7ff9f2ffd700 1 --2- 172.21.15.40:0/2168013181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9f40a5be0 0x7ff9f40aaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:11.212 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.206+0000 7ff9f2ffd700 10 monclient(hunting): get_auth_request con 0x7ff9f40a5be0 auth_method 0 2023-09-30T00:47:11.212 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.206+0000 7ff9f2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:11.213 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.206+0000 7ff9f2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:11.213 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.206+0000 7ff9f2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:11.213 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.207+0000 7ff9f37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:11.214 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.207+0000 7ff9f37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:11.214 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.207+0000 7ff9f37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:11.214 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.207+0000 7ff9f27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:11.214 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.207+0000 7ff9f27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:11.214 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.207+0000 7ff9f27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:11.214 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.207+0000 7ff9f2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:11.214 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.207+0000 7ff9f2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:11.214 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.207+0000 7ff9f2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:11.214 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.211+0000 7ff9f2ffd700 10 monclient(hunting): handle_auth_done global_id 4200 payload 824 2023-09-30T00:47:11.214 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.211+0000 7ff9f2ffd700 1 -- 172.21.15.40:0/2168013181 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9f40a63e0 msgr2=0x7ff9f4151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:11.214 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.211+0000 7ff9f2ffd700 1 --2- 172.21.15.40:0/2168013181 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9f40a63e0 0x7ff9f4151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:11.215 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.211+0000 7ff9f2ffd700 1 -- 172.21.15.40:0/2168013181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9f40a5fb0 msgr2=0x7ff9f413d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:11.215 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.211+0000 7ff9f2ffd700 1 --2- 172.21.15.40:0/2168013181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9f40a5fb0 0x7ff9f413d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:11.215 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.211+0000 7ff9f2ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:47:11.215 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.211+0000 7ff9f2ffd700 1 monclient: found mon.noname-a 2023-09-30T00:47:11.215 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.211+0000 7ff9f2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:11.215 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.211+0000 7ff9f2ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:11.215 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.211+0000 7ff9f2ffd700 1 -- 172.21.15.40:0/2168013181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e8009420 con 0x7ff9f40a5be0 2023-09-30T00:47:11.216 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.215+0000 7ff9f2ffd700 1 --2- 172.21.15.40:0/2168013181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9f40a5be0 0x7ff9f40aaba0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff9e400c590 tx=0x7ff9e4007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:11.216 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.215+0000 7ff9daffd700 10 client.?.objecter ms_handle_connect 0x7ff9f40a5be0 2023-09-30T00:47:11.216 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.215+0000 7ff9daffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:11.222 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.220+0000 7ff9daffd700 1 -- 172.21.15.40:0/2168013181 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4011590 con 0x7ff9f40a5be0 2023-09-30T00:47:11.222 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.220+0000 7ff9daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:11.222 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.220+0000 7ff9daffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:11.222 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.220+0000 7ff9daffd700 10 monclient: dump: 2023-09-30T00:47:11.223 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:11.223 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:11.223 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:11.223 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:11.223 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:11.223 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:11.224 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:11.224 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:11.224 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:11.224 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:11.224 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.220+0000 7ff9daffd700 10 monclient: _finish_auth 0 2023-09-30T00:47:11.224 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.220+0000 7ff9daffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:11.225 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9daffd700 1 -- 172.21.15.40:0/2168013181 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e4011710 con 0x7ff9f40a5be0 2023-09-30T00:47:11.225 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9f9282700 5 monclient: authenticate success, global_id 4200 2023-09-30T00:47:11.225 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9f9282700 10 monclient: _renew_subs 2023-09-30T00:47:11.225 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9f9282700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:11.225 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9f9282700 1 -- 172.21.15.40:0/2168013181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f40a8d70 con 0x7ff9f40a5be0 2023-09-30T00:47:11.225 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9daffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:11.226 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9daffd700 1 -- 172.21.15.40:0/2168013181 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4011ad0 con 0x7ff9f40a5be0 2023-09-30T00:47:11.226 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:11.226 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9daffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:11.226 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9daffd700 10 monclient: dump: 2023-09-30T00:47:11.226 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:11.227 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:11.227 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:11.227 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:11.227 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:11.227 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:11.227 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:11.228 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:11.228 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:11.228 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:11.228 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9f9282700 10 client.4200.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:11.228 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9f9282700 10 monclient: _renew_subs 2023-09-30T00:47:11.228 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9f9282700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:11.229 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9f9282700 1 -- 172.21.15.40:0/2168013181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f4151bb0 con 0x7ff9f40a5be0 2023-09-30T00:47:11.229 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9f9282700 1 librados: init done 2023-09-30T00:47:11.229 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.221+0000 7ff9f17fa700 4 set_mon_vals no callback set 2023-09-30T00:47:11.230 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.222+0000 7ff9f9282700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:11.230 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.222+0000 7ff9f9282700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:11.230 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.222+0000 7ff9f9282700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:11.230 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.222+0000 7ff9f9282700 1 -- 172.21.15.40:0/2168013181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f4064a00 con 0x7ff9f40a5be0 2023-09-30T00:47:11.418 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.415+0000 7ff9daffd700 1 -- 172.21.15.40:0/2168013181 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7ff9e401b3b0 con 0x7ff9f40a5be0 2023-09-30T00:47:11.418 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.416+0000 7ff9daffd700 1 --2- 172.21.15.40:0/2168013181 >> v2:172.21.15.135:6800/116426 conn(0x7ff9e0068800 0x7ff9e006acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:11.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.416+0000 7ff9daffd700 1 -- 172.21.15.40:0/2168013181 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7ff9e4082c50 con 0x7ff9f40a5be0 2023-09-30T00:47:11.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.416+0000 7ff9daffd700 10 client.4200.objecter ms_dispatch 0x7ff9f4001550 osd_map(10..10 src has 1..10) v4 2023-09-30T00:47:11.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.416+0000 7ff9daffd700 3 client.4200.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T00:47:11.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.416+0000 7ff9daffd700 3 client.4200.objecter handle_osd_map decoding full epoch 10 2023-09-30T00:47:11.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.416+0000 7ff9daffd700 20 client.4200.objecter dump_active .. 0 homeless 2023-09-30T00:47:11.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.416+0000 7ff9f27fc700 1 --2- 172.21.15.40:0/2168013181 >> v2:172.21.15.135:6800/116426 conn(0x7ff9e0068800 0x7ff9e006acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:11.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.416+0000 7ff9f27fc700 10 monclient: get_auth_request con 0x7ff9e0068800 auth_method 0 2023-09-30T00:47:11.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.417+0000 7ff9daffd700 10 client.4200.objecter ms_handle_connect 0x7ff9e0068800 2023-09-30T00:47:11.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.417+0000 7ff9f27fc700 1 --2- 172.21.15.40:0/2168013181 >> v2:172.21.15.135:6800/116426 conn(0x7ff9e0068800 0x7ff9e006acb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff9f413c310 tx=0x7ff9e8017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:11.648 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.647+0000 7ff9daffd700 1 -- 172.21.15.40:0/2168013181 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff9e4050f60 con 0x7ff9f40a5be0 2023-09-30T00:47:11.648 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.648+0000 7ff9daffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:11.648 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.648+0000 7ff9daffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:11.918 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.917+0000 7ff9f9282700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:11.918 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.917+0000 7ff9f9282700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:11.918 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.917+0000 7ff9f9282700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:11.918 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:11.917+0000 7ff9f9282700 1 -- 172.21.15.40:0/2168013181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9f40aa510 con 0x7ff9f40a5be0 2023-09-30T00:47:12.020 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.019+0000 7ff9daffd700 1 -- 172.21.15.40:0/2168013181 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7ff9e4017020 con 0x7ff9f40a5be0 2023-09-30T00:47:12.020 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.019+0000 7ff9daffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:12.020 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.019+0000 7ff9daffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:12.021 INFO:teuthology.misc.health.smithi040.stdout: 2023-09-30T00:47:12.021 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":10,"fsid":"238dba7c-9190-48bd-bb67-322ebc9ecbd0","created":"2023-09-30T00:46:45.822318+0000","modified":"2023-09-30T00:46:52.657791+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:46:52.657791+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"66e1da65-310e-4463-8aa0-431896832e64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d6dd269f-2eda-4ef7-9da1-2ed5b16291b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f0de27d-3674-4ff4-8451-4fcc90786c98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d18ec2f5-7e8f-4732-a523-adc495115cbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e2465fe-bb27-47bd-8ddf-f334cac6041a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7ea449db-9b22-48b8-a61b-1937eeeb6c8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c79b4829-6393-4f9a-a9f8-d2aacb505ae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:47:12.023 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.022+0000 7ff9f9282700 10 librados: watch_flush enter 2023-09-30T00:47:12.023 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.022+0000 7ff9f9282700 10 librados: watch_flush exit 2023-09-30T00:47:12.023 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.022+0000 7ff9f9282700 20 client.4200.objecter shutdown clearing up homeless session... 2023-09-30T00:47:12.023 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.022+0000 7ff9f9282700 10 client.4200.objecter successfully canceled tick 2023-09-30T00:47:12.023 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.023+0000 7ff9f9282700 1 -- 172.21.15.40:0/2168013181 >> v2:172.21.15.135:6800/116426 conn(0x7ff9e0068800 msgr2=0x7ff9e006acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:12.024 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.023+0000 7ff9f9282700 1 --2- 172.21.15.40:0/2168013181 >> v2:172.21.15.135:6800/116426 conn(0x7ff9e0068800 0x7ff9e006acb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff9f413c310 tx=0x7ff9e8017000 comp rx=0 tx=0).stop 2023-09-30T00:47:12.024 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.023+0000 7ff9f9282700 10 monclient: shutdown 2023-09-30T00:47:12.024 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.023+0000 7ff9f9282700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:12.024 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.023+0000 7ff9f9282700 1 -- 172.21.15.40:0/2168013181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9f40a5be0 msgr2=0x7ff9f40aaba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:12.024 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.023+0000 7ff9f9282700 1 --2- 172.21.15.40:0/2168013181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9f40a5be0 0x7ff9f40aaba0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff9e400c590 tx=0x7ff9e4007ec0 comp rx=0 tx=0).stop 2023-09-30T00:47:12.024 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.024+0000 7ff9f9282700 1 -- 172.21.15.40:0/2168013181 shutdown_connections 2023-09-30T00:47:12.025 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.024+0000 7ff9f9282700 1 --2- 172.21.15.40:0/2168013181 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff9f40a63e0 0x7ff9f4151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:12.025 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.024+0000 7ff9f9282700 1 --2- 172.21.15.40:0/2168013181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9f40a5be0 0x7ff9f40aaba0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:12.025 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.024+0000 7ff9f9282700 1 --2- 172.21.15.40:0/2168013181 >> v2:172.21.15.135:6800/116426 conn(0x7ff9e0068800 0x7ff9e006acb0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:12.025 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.024+0000 7ff9f9282700 1 --2- 172.21.15.40:0/2168013181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff9f40a5fb0 0x7ff9f413d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:12.025 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.024+0000 7ff9f9282700 1 -- 172.21.15.40:0/2168013181 >> 172.21.15.40:0/2168013181 conn(0x7ff9f4063280 msgr2=0x7ff9f4063650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:12.025 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.024+0000 7ff9f9282700 1 -- 172.21.15.40:0/2168013181 shutdown_connections 2023-09-30T00:47:12.025 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.024+0000 7ff9f9282700 1 -- 172.21.15.40:0/2168013181 wait complete. 2023-09-30T00:47:12.025 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:12.024+0000 7ff9f9282700 1 librados: shutdown 2023-09-30T00:47:12.041 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T00:47:13.434 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:20.537 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:13.438 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:20.567 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:13.469 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:20.598 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:13.524 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:20.646 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:13.949 INFO:tasks.ceph.mon.a.smithi040.stderr:2023-09-30T00:47:13.945+0000 13c1c700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T00:47:15.676 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:22.784 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:15.875 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:22.976 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:16.027 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:23.141 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:16.223 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:23.324 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:18.847 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:47:18.985 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.984+0000 7f42349f7700 10 monclient: get_monmap_and_config 2023-09-30T00:47:18.985 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.984+0000 7f42349f7700 10 monclient: build_initial_monmap 2023-09-30T00:47:18.986 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.984+0000 7f42349f7700 10 monclient: monmap: 2023-09-30T00:47:18.986 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:47:18.986 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:18.986 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:47:18.985749+0000 2023-09-30T00:47:18.986 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:47:18.985749+0000 2023-09-30T00:47:18.986 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:18.986 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:18.986 INFO:teuthology.misc.health.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:18.987 INFO:teuthology.misc.health.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:18.987 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:18.987 INFO:teuthology.misc.health.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:18.987 INFO:teuthology.misc.health.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:18.987 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:18.987 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.985+0000 7f42349f7700 1 Processor -- start 2023-09-30T00:47:18.987 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.985+0000 7f42349f7700 1 -- start start 2023-09-30T00:47:18.987 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.985+0000 7f42349f7700 10 monclient: init 2023-09-30T00:47:18.988 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.986+0000 7f42349f7700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:18.988 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.986+0000 7f42349f7700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-09-30T00:47:18.988 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.986+0000 7f42349f7700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f423006b740 0x7f42300af960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:18.988 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.986+0000 7f42349f7700 10 monclient(hunting): picked mon.noname-c con 0x7f423006b740 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:18.988 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.986+0000 7f42349f7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42300afea0 0x7f42300b0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:18.988 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.986+0000 7f42349f7700 10 monclient(hunting): picked mon.noname-b con 0x7f42300afea0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:18.988 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.986+0000 7f42349f7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4230146130 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:18.989 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.986+0000 7f42349f7700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423006bb10 con 0x7f4230146130 2023-09-30T00:47:18.989 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.986+0000 7f42349f7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:18.989 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.986+0000 7f42349f7700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f423006d0b0 con 0x7f423006b740 2023-09-30T00:47:18.989 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.986+0000 7f42349f7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:18.989 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.986+0000 7f42349f7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423006cdd0 con 0x7f42300afea0 2023-09-30T00:47:18.989 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.986+0000 7f42349f7700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:18.989 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.986+0000 7f42349f7700 10 monclient(hunting): authenticate will time out at 5300.287849s 2023-09-30T00:47:18.989 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.987+0000 7f422dd9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42300afea0 0x7f42300b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:18.990 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.988+0000 7f422dd9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42300afea0 0x7f42300b0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:37644/0 (socket says 172.21.15.40:37644) 2023-09-30T00:47:18.990 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.988+0000 7f422dd9b700 1 -- 172.21.15.40:0/913733091 learned_addr learned my addr 172.21.15.40:0/913733091 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:18.990 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.988+0000 7f422dd9b700 10 monclient(hunting): get_auth_request con 0x7f42300afea0 auth_method 0 2023-09-30T00:47:18.990 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.988+0000 7f422dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:18.990 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.988+0000 7f422dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:18.990 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.988+0000 7f422dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:18.990 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.988+0000 7f422e59c700 1 --2- 172.21.15.40:0/913733091 >> v2:172.21.15.135:3300/0 conn(0x7f423006b740 0x7f42300af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:18.991 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.988+0000 7f422dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:18.991 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.988+0000 7f422dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:18.991 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.988+0000 7f422dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:18.991 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.988+0000 7f422e59c700 10 monclient(hunting): get_auth_request con 0x7f423006b740 auth_method 0 2023-09-30T00:47:18.991 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.988+0000 7f422e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:18.991 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.988+0000 7f422e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:18.991 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.988+0000 7f422e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:18.992 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.990+0000 7f422e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:18.992 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.990+0000 7f422e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:18.992 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.990+0000 7f422e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:18.992 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.991+0000 7f422dd9b700 10 monclient(hunting): handle_auth_done global_id 4172 payload 274 2023-09-30T00:47:18.992 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.991+0000 7f422dd9b700 1 -- 172.21.15.40:0/913733091 >> v2:172.21.15.135:3300/0 conn(0x7f423006b740 msgr2=0x7f42300af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:18.992 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.991+0000 7f422dd9b700 1 --2- 172.21.15.40:0/913733091 >> v2:172.21.15.135:3300/0 conn(0x7f423006b740 0x7f42300af960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:18.992 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.991+0000 7f422dd9b700 1 -- 172.21.15.40:0/913733091 >> v1:172.21.15.135:6789/0 conn(0x7f4230146130 legacy=0x7f42301465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:18.992 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.991+0000 7f422dd9b700 10 monclient: _finish_hunting 0 2023-09-30T00:47:18.993 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.991+0000 7f422dd9b700 1 monclient: found mon.noname-b 2023-09-30T00:47:18.993 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.991+0000 7f422dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:18.993 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.991+0000 7f422dd9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:47:18.993 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.991+0000 7f422dd9b700 1 -- 172.21.15.40:0/913733091 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423006f3a0 con 0x7f42300afea0 2023-09-30T00:47:18.994 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.993+0000 7f422dd9b700 1 --2- 172.21.15.40:0/913733091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42300afea0 0x7f42300b0270 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f42240099b0 tx=0x7f4224009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e36d34429f52de24 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:18.999 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f422d59a700 1 -- 172.21.15.40:0/913733091 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4224009ea0 con 0x7f42300afea0 2023-09-30T00:47:18.999 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f422d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:18.999 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f422d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:18.999 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f422d59a700 10 monclient: dump: 2023-09-30T00:47:18.999 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:18.999 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:18.999 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:18.999 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:19.000 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:19.000 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:19.000 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:19.000 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:19.000 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:19.000 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:19.000 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f422d59a700 10 monclient: _finish_auth 0 2023-09-30T00:47:19.000 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f422d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:19.000 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f422d59a700 1 -- 172.21.15.40:0/913733091 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422400b820 con 0x7f42300afea0 2023-09-30T00:47:19.000 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f42349f7700 5 monclient: authenticate success, global_id 4172 2023-09-30T00:47:19.000 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f42349f7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:19.000 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f422d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:19.000 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f422d59a700 1 -- 172.21.15.40:0/913733091 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422400bbe0 con 0x7f42300afea0 2023-09-30T00:47:19.001 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f42349f7700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:19.001 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f42349f7700 4 set_mon_vals no callback set 2023-09-30T00:47:19.001 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f422d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:19.001 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f422d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:19.001 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.998+0000 7f422d59a700 10 monclient: dump: 2023-09-30T00:47:19.001 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:19.001 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:19.001 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:19.001 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:19.001 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:19.001 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:19.002 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:19.002 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:19.002 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:19.002 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:19.003 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.999+0000 7f42349f7700 10 monclient: shutdown 2023-09-30T00:47:19.003 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.999+0000 7f42349f7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:19.003 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.999+0000 7f42349f7700 1 -- 172.21.15.40:0/913733091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42300afea0 msgr2=0x7f42300b0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:19.003 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.999+0000 7f42349f7700 1 --2- 172.21.15.40:0/913733091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42300afea0 0x7f42300b0270 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f42240099b0 tx=0x7f4224009250 comp rx=0 tx=0).stop 2023-09-30T00:47:19.003 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.999+0000 7f42349f7700 1 -- 172.21.15.40:0/913733091 shutdown_connections 2023-09-30T00:47:19.003 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.999+0000 7f42349f7700 1 --2- 172.21.15.40:0/913733091 >> v2:172.21.15.135:3300/0 conn(0x7f423006b740 0x7f42300af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:19.003 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.999+0000 7f42349f7700 1 --2- 172.21.15.40:0/913733091 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42300afea0 0x7f42300b0270 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:19.003 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.999+0000 7f42349f7700 1 -- 172.21.15.40:0/913733091 >> 172.21.15.40:0/913733091 conn(0x7f42300b4aa0 msgr2=0x7f42300b4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:19.003 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.999+0000 7f42349f7700 1 -- 172.21.15.40:0/913733091 shutdown_connections 2023-09-30T00:47:19.003 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:18.999+0000 7f42349f7700 1 -- 172.21.15.40:0/913733091 wait complete. 2023-09-30T00:47:19.003 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.000+0000 7f42349f7700 10 monclient: build_initial_monmap 2023-09-30T00:47:19.004 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.000+0000 7f42349f7700 10 monclient: monmap: 2023-09-30T00:47:19.004 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:47:19.004 INFO:teuthology.misc.health.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:19.004 INFO:teuthology.misc.health.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:19.004 INFO:teuthology.misc.health.smithi040.stderr:created 0.000000 2023-09-30T00:47:19.004 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:19.004 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:19.004 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:19.004 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:19.004 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:19.004 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:19.004 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.000+0000 7f42349f7700 1 librados: starting msgr at 2023-09-30T00:47:19.005 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.000+0000 7f42349f7700 1 librados: starting objecter 2023-09-30T00:47:19.005 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.000+0000 7f42349f7700 1 Processor -- start 2023-09-30T00:47:19.005 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 1 -- start start 2023-09-30T00:47:19.005 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 1 librados: setting wanted keys 2023-09-30T00:47:19.005 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 1 librados: calling monclient init 2023-09-30T00:47:19.005 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 10 monclient: init 2023-09-30T00:47:19.005 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:19.005 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:47:19.005 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f423006b740 0x7f4230143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:19.005 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 10 monclient(hunting): picked mon.noname-b con 0x7f423006b740 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:19.005 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42300afea0 0x7f42300b86f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:19.005 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 10 monclient(hunting): picked mon.noname-c con 0x7f42300afea0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:19.006 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4230146130 0x7f42300b7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:19.006 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 10 monclient(hunting): picked mon.noname-a con 0x7f4230146130 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:19.006 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:19.006 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42300b9040 con 0x7f4230146130 2023-09-30T00:47:19.006 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:19.006 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42300b8c30 con 0x7f423006b740 2023-09-30T00:47:19.006 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:19.006 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4230145d50 con 0x7f42300afea0 2023-09-30T00:47:19.006 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:19.006 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.001+0000 7f42349f7700 10 monclient(hunting): authenticate will time out at 5300.302940s 2023-09-30T00:47:19.006 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.002+0000 7f422dd9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42300afea0 0x7f42300b86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:19.007 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.002+0000 7f422ed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4230146130 0x7f42300b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:19.007 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.003+0000 7f422dd9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42300afea0 0x7f42300b86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:37646/0 (socket says 172.21.15.40:37646) 2023-09-30T00:47:19.007 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.003+0000 7f422dd9b700 1 -- 172.21.15.40:0/1723247024 learned_addr learned my addr 172.21.15.40:0/1723247024 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:19.008 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.003+0000 7f422dd9b700 10 monclient(hunting): get_auth_request con 0x7f42300afea0 auth_method 0 2023-09-30T00:47:19.008 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.003+0000 7f422dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:19.008 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.003+0000 7f422dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:19.008 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.003+0000 7f422dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:19.008 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.003+0000 7f422ed9d700 10 monclient(hunting): get_auth_request con 0x7f4230146130 auth_method 0 2023-09-30T00:47:19.008 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.003+0000 7f422ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:19.008 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.003+0000 7f422ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:19.008 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.003+0000 7f422ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:19.008 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.003+0000 7f422dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:19.008 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.003+0000 7f422dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:19.008 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.003+0000 7f422dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:19.008 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.004+0000 7f422ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:19.008 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.004+0000 7f422ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:19.009 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.004+0000 7f422ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:19.009 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.006+0000 7f422dd9b700 10 monclient(hunting): handle_auth_done global_id 4175 payload 824 2023-09-30T00:47:19.009 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.006+0000 7f422dd9b700 1 -- 172.21.15.40:0/1723247024 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f423006b740 msgr2=0x7f4230143ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:19.009 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.006+0000 7f422dd9b700 1 --2- 172.21.15.40:0/1723247024 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f423006b740 0x7f4230143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:19.009 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.007+0000 7f422dd9b700 1 -- 172.21.15.40:0/1723247024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4230146130 msgr2=0x7f42300b7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:19.009 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.007+0000 7f422dd9b700 1 --2- 172.21.15.40:0/1723247024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4230146130 0x7f42300b7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:19.009 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.007+0000 7f422dd9b700 10 monclient: _finish_hunting 0 2023-09-30T00:47:19.009 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.007+0000 7f422dd9b700 1 monclient: found mon.noname-c 2023-09-30T00:47:19.009 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.007+0000 7f422dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:19.009 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.007+0000 7f422dd9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:47:19.009 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.007+0000 7f422dd9b700 1 -- 172.21.15.40:0/1723247024 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42240094f0 con 0x7f42300afea0 2023-09-30T00:47:19.010 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.010+0000 7f422dd9b700 1 --2- 172.21.15.40:0/1723247024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42300afea0 0x7f42300b86f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f423006bca0 tx=0x7f4224016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:19.010 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.010+0000 7f421e7fc700 10 client.?.objecter ms_handle_connect 0x7f42300afea0 2023-09-30T00:47:19.010 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.010+0000 7f421e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:19.015 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.014+0000 7f421e7fc700 1 -- 172.21.15.40:0/1723247024 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4224006b60 con 0x7f42300afea0 2023-09-30T00:47:19.016 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f421e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:19.016 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f421e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:19.016 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f421e7fc700 10 monclient: dump: 2023-09-30T00:47:19.016 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:19.017 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:19.017 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:19.017 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:19.017 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:19.017 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:19.017 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:19.018 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:19.018 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:19.018 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:19.018 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f421e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:47:19.018 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f421e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:19.019 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f421e7fc700 1 -- 172.21.15.40:0/1723247024 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4224020070 con 0x7f42300afea0 2023-09-30T00:47:19.019 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f42349f7700 5 monclient: authenticate success, global_id 4175 2023-09-30T00:47:19.019 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f42349f7700 10 monclient: _renew_subs 2023-09-30T00:47:19.019 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f42349f7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:19.019 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f42349f7700 1 -- 172.21.15.40:0/1723247024 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42300b7b80 con 0x7f42300afea0 2023-09-30T00:47:19.019 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f421e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:19.020 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f421e7fc700 1 -- 172.21.15.40:0/1723247024 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4224011d60 con 0x7f42300afea0 2023-09-30T00:47:19.020 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f421e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:19.020 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f421e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:19.020 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f421e7fc700 10 monclient: dump: 2023-09-30T00:47:19.020 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:19.020 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:19.021 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:19.021 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:19.021 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:19.021 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:19.021 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:19.021 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:19.022 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:19.022 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:19.022 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f422cd99700 4 set_mon_vals no callback set 2023-09-30T00:47:19.022 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f42349f7700 10 client.4175.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:19.022 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f42349f7700 10 monclient: _renew_subs 2023-09-30T00:47:19.022 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f42349f7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:19.022 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f42349f7700 1 -- 172.21.15.40:0/1723247024 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42300b7040 con 0x7f42300afea0 2023-09-30T00:47:19.022 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.015+0000 7f42349f7700 1 librados: init done 2023-09-30T00:47:19.023 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.016+0000 7f42349f7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:19.023 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.016+0000 7f42349f7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:19.023 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.016+0000 7f42349f7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:19.023 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.016+0000 7f42349f7700 1 -- 172.21.15.40:0/1723247024 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42300b91b0 con 0x7f42300afea0 2023-09-30T00:47:19.046 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.044+0000 7f421e7fc700 1 -- 172.21.15.40:0/1723247024 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f4224022030 con 0x7f42300afea0 2023-09-30T00:47:19.046 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.045+0000 7f421e7fc700 1 --2- 172.21.15.40:0/1723247024 >> v2:172.21.15.135:6800/116426 conn(0x7f4210068880 0x7f421006ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:19.046 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.045+0000 7f421e7fc700 1 -- 172.21.15.40:0/1723247024 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f4224085540 con 0x7f42300afea0 2023-09-30T00:47:19.046 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.045+0000 7f421e7fc700 10 client.4175.objecter ms_dispatch 0x7f4230001550 osd_map(10..10 src has 1..10) v4 2023-09-30T00:47:19.046 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.045+0000 7f421e7fc700 3 client.4175.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T00:47:19.046 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.045+0000 7f421e7fc700 3 client.4175.objecter handle_osd_map decoding full epoch 10 2023-09-30T00:47:19.046 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.045+0000 7f421e7fc700 20 client.4175.objecter dump_active .. 0 homeless 2023-09-30T00:47:19.046 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.045+0000 7f422e59c700 1 --2- 172.21.15.40:0/1723247024 >> v2:172.21.15.135:6800/116426 conn(0x7f4210068880 0x7f421006ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:19.047 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.045+0000 7f422e59c700 10 monclient: get_auth_request con 0x7f4210068880 auth_method 0 2023-09-30T00:47:19.047 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.045+0000 7f421e7fc700 10 client.4175.objecter ms_handle_connect 0x7f4210068880 2023-09-30T00:47:19.047 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.045+0000 7f422e59c700 1 --2- 172.21.15.40:0/1723247024 >> v2:172.21.15.135:6800/116426 conn(0x7f4210068880 0x7f421006ad30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4220009bb0 tx=0x7f42200091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:19.382 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.381+0000 7f421e7fc700 1 -- 172.21.15.40:0/1723247024 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4224053850 con 0x7f42300afea0 2023-09-30T00:47:19.382 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.381+0000 7f421e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:19.382 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.381+0000 7f421e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:19.385 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:26.488 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:19.391 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:26.495 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:19.524 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:26.652 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:19.529 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:26.658 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:19.554 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:26.676 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:19.561 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:26.683 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:19.638 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.637+0000 7f42349f7700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:19.638 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.637+0000 7f42349f7700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:19.639 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.637+0000 7f42349f7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:19.639 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.637+0000 7f42349f7700 1 -- 172.21.15.40:0/1723247024 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42300b91b0 con 0x7f42300afea0 2023-09-30T00:47:19.674 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.673+0000 7f421e7fc700 1 -- 172.21.15.40:0/1723247024 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f4224056e70 con 0x7f42300afea0 2023-09-30T00:47:19.674 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.673+0000 7f421e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:19.674 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.673+0000 7f421e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:19.674 INFO:teuthology.misc.health.smithi040.stdout: 2023-09-30T00:47:19.674 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":10,"fsid":"238dba7c-9190-48bd-bb67-322ebc9ecbd0","created":"2023-09-30T00:46:45.822318+0000","modified":"2023-09-30T00:46:52.657791+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:46:52.657791+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"66e1da65-310e-4463-8aa0-431896832e64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d6dd269f-2eda-4ef7-9da1-2ed5b16291b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f0de27d-3674-4ff4-8451-4fcc90786c98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d18ec2f5-7e8f-4732-a523-adc495115cbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e2465fe-bb27-47bd-8ddf-f334cac6041a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7ea449db-9b22-48b8-a61b-1937eeeb6c8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c79b4829-6393-4f9a-a9f8-d2aacb505ae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:47:19.677 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.676+0000 7f42349f7700 10 librados: watch_flush enter 2023-09-30T00:47:19.677 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.676+0000 7f42349f7700 10 librados: watch_flush exit 2023-09-30T00:47:19.677 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.676+0000 7f42349f7700 20 client.4175.objecter shutdown clearing up homeless session... 2023-09-30T00:47:19.677 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.676+0000 7f42349f7700 10 client.4175.objecter successfully canceled tick 2023-09-30T00:47:19.677 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.676+0000 7f42349f7700 1 -- 172.21.15.40:0/1723247024 >> v2:172.21.15.135:6800/116426 conn(0x7f4210068880 msgr2=0x7f421006ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:19.677 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.676+0000 7f42349f7700 1 --2- 172.21.15.40:0/1723247024 >> v2:172.21.15.135:6800/116426 conn(0x7f4210068880 0x7f421006ad30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4220009bb0 tx=0x7f42200091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:19.677 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.676+0000 7f42349f7700 10 monclient: shutdown 2023-09-30T00:47:19.677 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.676+0000 7f42349f7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:19.677 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.676+0000 7f42349f7700 1 -- 172.21.15.40:0/1723247024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42300afea0 msgr2=0x7f42300b86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:19.678 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.676+0000 7f42349f7700 1 --2- 172.21.15.40:0/1723247024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42300afea0 0x7f42300b86f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f423006bca0 tx=0x7f4224016000 comp rx=0 tx=0).stop 2023-09-30T00:47:19.679 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.677+0000 7f42349f7700 1 -- 172.21.15.40:0/1723247024 shutdown_connections 2023-09-30T00:47:19.679 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.677+0000 7f42349f7700 1 --2- 172.21.15.40:0/1723247024 >> v2:172.21.15.135:6800/116426 conn(0x7f4210068880 0x7f421006ad30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:19.679 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.677+0000 7f42349f7700 1 --2- 172.21.15.40:0/1723247024 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f423006b740 0x7f4230143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:19.679 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.677+0000 7f42349f7700 1 --2- 172.21.15.40:0/1723247024 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4230146130 0x7f42300b7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:19.680 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.677+0000 7f42349f7700 1 --2- 172.21.15.40:0/1723247024 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f42300afea0 0x7f42300b86f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:19.680 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.677+0000 7f42349f7700 1 -- 172.21.15.40:0/1723247024 >> 172.21.15.40:0/1723247024 conn(0x7f42300b4aa0 msgr2=0x7f42300b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:19.681 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.681+0000 7f42349f7700 1 -- 172.21.15.40:0/1723247024 shutdown_connections 2023-09-30T00:47:19.681 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.681+0000 7f42349f7700 1 -- 172.21.15.40:0/1723247024 wait complete. 2023-09-30T00:47:19.681 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:19.681+0000 7f42349f7700 1 librados: shutdown 2023-09-30T00:47:19.697 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T00:47:19.886 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:27.015 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:19.892 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:27.020 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:22.727 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:29.829 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:22.730 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:29.838 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:22.733 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:29.835 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:22.737 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:29.844 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:22.915 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:30.029 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:22.921 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:30.035 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:23.164 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:30.265 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:23.171 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:30.271 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:24.058 INFO:tasks.ceph.osd.6.smithi135.stderr:==00:00:00:31.161 117133== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:24.378 INFO:tasks.ceph.osd.4.smithi135.stderr:==00:00:00:31.507 117131== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:24.608 INFO:tasks.ceph.osd.7.smithi135.stderr:==00:00:00:31.730 117132== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:24.829 INFO:tasks.ceph.osd.5.smithi135.stderr:==00:00:00:31.957 117130== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:26.504 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:47:26.664 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.663+0000 7f75927a2700 10 monclient: get_monmap_and_config 2023-09-30T00:47:26.664 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.663+0000 7f75927a2700 10 monclient: build_initial_monmap 2023-09-30T00:47:26.664 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.663+0000 7f75927a2700 10 monclient: monmap: 2023-09-30T00:47:26.664 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:47:26.664 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:26.665 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:47:26.664612+0000 2023-09-30T00:47:26.665 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:47:26.664612+0000 2023-09-30T00:47:26.665 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:26.665 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:26.665 INFO:teuthology.misc.health.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:26.665 INFO:teuthology.misc.health.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:26.665 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:26.665 INFO:teuthology.misc.health.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:26.665 INFO:teuthology.misc.health.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:26.665 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:26.666 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.664+0000 7f75927a2700 1 Processor -- start 2023-09-30T00:47:26.666 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.664+0000 7f75927a2700 1 -- start start 2023-09-30T00:47:26.666 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.664+0000 7f75927a2700 10 monclient: init 2023-09-30T00:47:26.666 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.665+0000 7f75927a2700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:26.666 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.665+0000 7f75927a2700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-09-30T00:47:26.666 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.665+0000 7f75927a2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f758c0a5be0 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:26.667 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.665+0000 7f75927a2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f758c0a5fb0 0x7f758c141d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:26.667 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.665+0000 7f75927a2700 10 monclient(hunting): picked mon.noname-a con 0x7f758c0a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:26.667 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.665+0000 7f75927a2700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f758c0a63e0 0x7f758c142240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:26.667 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.665+0000 7f75927a2700 10 monclient(hunting): picked mon.noname-c con 0x7f758c0a63e0 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:26.667 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.665+0000 7f75927a2700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f758c0a7750 con 0x7f758c0a5be0 2023-09-30T00:47:26.667 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.665+0000 7f75927a2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:26.667 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.665+0000 7f75927a2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f758c0a59d0 con 0x7f758c0a5fb0 2023-09-30T00:47:26.667 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.665+0000 7f75927a2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:26.667 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.665+0000 7f75927a2700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f758c0a56f0 con 0x7f758c0a63e0 2023-09-30T00:47:26.667 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.665+0000 7f75927a2700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:26.667 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.665+0000 7f75927a2700 10 monclient(hunting): authenticate will time out at 5307.966395s 2023-09-30T00:47:26.668 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.666+0000 7f758b7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f758c0a5fb0 0x7f758c141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:26.668 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.666+0000 7f7590d3f700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f758c0a63e0 0x7f758c142240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:26.668 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.666+0000 7f758bfff700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f758c0a5be0 0x7f758c13eba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:56334/0 (socket says 172.21.15.40:56334) 2023-09-30T00:47:26.669 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.666+0000 7f758bfff700 1 -- 172.21.15.40:0/2334515613 learned_addr learned my addr 172.21.15.40:0/2334515613 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:26.669 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.666+0000 7f758b7fe700 10 monclient(hunting): get_auth_request con 0x7f758c0a5fb0 auth_method 0 2023-09-30T00:47:26.669 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.666+0000 7f758b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:26.669 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.666+0000 7f758b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:26.669 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.666+0000 7f758b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:26.669 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.666+0000 7f7590d3f700 10 monclient(hunting): get_auth_request con 0x7f758c0a63e0 auth_method 0 2023-09-30T00:47:26.669 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.666+0000 7f7590d3f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:26.669 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.666+0000 7f7590d3f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:26.669 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.666+0000 7f7590d3f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:26.669 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.667+0000 7f7590d3f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:26.669 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.667+0000 7f7590d3f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:26.669 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.667+0000 7f7590d3f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:26.669 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.667+0000 7f758b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:26.670 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.667+0000 7f758b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:26.670 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.667+0000 7f758b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:26.670 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.670+0000 7f7590d3f700 10 monclient(hunting): handle_auth_done global_id 4234 payload 274 2023-09-30T00:47:26.670 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.670+0000 7f7590d3f700 1 -- 172.21.15.40:0/2334515613 >> v2:172.21.15.40:3300/0 conn(0x7f758c0a5fb0 msgr2=0x7f758c141d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:26.671 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.670+0000 7f7590d3f700 1 --2- 172.21.15.40:0/2334515613 >> v2:172.21.15.40:3300/0 conn(0x7f758c0a5fb0 0x7f758c141d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:26.671 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.670+0000 7f7590d3f700 1 -- 172.21.15.40:0/2334515613 >> v1:172.21.15.135:6789/0 conn(0x7f758c0a5be0 legacy=0x7f758c13eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:26.671 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.670+0000 7f7590d3f700 10 monclient: _finish_hunting 0 2023-09-30T00:47:26.671 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.670+0000 7f7590d3f700 1 monclient: found mon.noname-c 2023-09-30T00:47:26.671 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.670+0000 7f7590d3f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:26.671 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.670+0000 7f7590d3f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:47:26.671 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.670+0000 7f7590d3f700 1 -- 172.21.15.40:0/2334515613 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758c13f2b0 con 0x7f758c0a63e0 2023-09-30T00:47:26.675 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.674+0000 7f7590d3f700 1 --2- 172.21.15.40:0/2334515613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f758c0a63e0 0x7f758c142240 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f758c0b0dc0 tx=0x7f7584009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b712eac8199efeca server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:26.679 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.678+0000 7f758affd700 1 -- 172.21.15.40:0/2334515613 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758400e070 con 0x7f758c0a63e0 2023-09-30T00:47:26.679 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f758affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:26.680 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f758affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:26.680 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f758affd700 10 monclient: dump: 2023-09-30T00:47:26.680 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:26.680 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:26.680 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:26.680 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:26.680 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:26.680 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:26.680 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:26.680 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:26.680 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:26.680 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:26.680 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f758affd700 10 monclient: _finish_auth 0 2023-09-30T00:47:26.681 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f758affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:26.681 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f758affd700 1 -- 172.21.15.40:0/2334515613 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758400bb60 con 0x7f758c0a63e0 2023-09-30T00:47:26.681 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f75927a2700 5 monclient: authenticate success, global_id 4234 2023-09-30T00:47:26.681 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f75927a2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:26.681 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f758affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:26.681 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f75927a2700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:26.681 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f758affd700 1 -- 172.21.15.40:0/2334515613 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7584012430 con 0x7f758c0a63e0 2023-09-30T00:47:26.681 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f75927a2700 4 set_mon_vals no callback set 2023-09-30T00:47:26.681 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f758affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:26.681 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f758affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:26.681 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f758affd700 10 monclient: dump: 2023-09-30T00:47:26.681 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:26.681 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:26.682 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:26.682 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:26.682 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:26.682 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:26.682 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:26.682 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:26.682 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:26.682 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:26.684 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f75927a2700 10 monclient: shutdown 2023-09-30T00:47:26.684 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f75927a2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:26.685 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f75927a2700 1 -- 172.21.15.40:0/2334515613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f758c0a63e0 msgr2=0x7f758c142240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:26.685 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f75927a2700 1 --2- 172.21.15.40:0/2334515613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f758c0a63e0 0x7f758c142240 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f758c0b0dc0 tx=0x7f7584009780 comp rx=0 tx=0).stop 2023-09-30T00:47:26.685 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f75927a2700 1 -- 172.21.15.40:0/2334515613 shutdown_connections 2023-09-30T00:47:26.685 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f75927a2700 1 --2- 172.21.15.40:0/2334515613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f758c0a63e0 0x7f758c142240 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:26.685 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f75927a2700 1 --2- 172.21.15.40:0/2334515613 >> v2:172.21.15.40:3300/0 conn(0x7f758c0a5fb0 0x7f758c141d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:26.685 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f75927a2700 1 -- 172.21.15.40:0/2334515613 >> 172.21.15.40:0/2334515613 conn(0x7f758c063280 msgr2=0x7f758c063680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:26.685 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f75927a2700 1 -- 172.21.15.40:0/2334515613 shutdown_connections 2023-09-30T00:47:26.685 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.679+0000 7f75927a2700 1 -- 172.21.15.40:0/2334515613 wait complete. 2023-09-30T00:47:26.685 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.680+0000 7f75927a2700 10 monclient: build_initial_monmap 2023-09-30T00:47:26.685 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.680+0000 7f75927a2700 10 monclient: monmap: 2023-09-30T00:47:26.685 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:47:26.686 INFO:teuthology.misc.health.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:26.686 INFO:teuthology.misc.health.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:26.686 INFO:teuthology.misc.health.smithi040.stderr:created 0.000000 2023-09-30T00:47:26.686 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:26.686 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:26.686 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:26.686 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:26.686 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:26.686 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:26.686 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.681+0000 7f75927a2700 1 librados: starting msgr at 2023-09-30T00:47:26.686 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.681+0000 7f75927a2700 1 librados: starting objecter 2023-09-30T00:47:26.687 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.681+0000 7f75927a2700 1 Processor -- start 2023-09-30T00:47:26.687 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.681+0000 7f75927a2700 1 -- start start 2023-09-30T00:47:26.687 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.681+0000 7f75927a2700 1 librados: setting wanted keys 2023-09-30T00:47:26.687 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.681+0000 7f75927a2700 1 librados: calling monclient init 2023-09-30T00:47:26.687 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.681+0000 7f75927a2700 10 monclient: init 2023-09-30T00:47:26.687 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:26.687 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:47:26.687 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f758c0a5be0 0x7f758c151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:26.687 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 10 monclient(hunting): picked mon.noname-c con 0x7f758c0a5be0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:26.687 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c0a5fb0 0x7f758c0aa940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:26.687 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 10 monclient(hunting): picked mon.noname-a con 0x7f758c0a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:26.687 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f758c0a63e0 0x7f758c0ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:26.688 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 10 monclient(hunting): picked mon.noname-b con 0x7f758c0a63e0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:26.688 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:26.688 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c0ab230 con 0x7f758c0a5fb0 2023-09-30T00:47:26.688 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:26.688 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c0aae80 con 0x7f758c0a63e0 2023-09-30T00:47:26.688 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:26.688 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758c1524a0 con 0x7f758c0a5be0 2023-09-30T00:47:26.688 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:26.688 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.682+0000 7f75927a2700 10 monclient(hunting): authenticate will time out at 5307.983462s 2023-09-30T00:47:26.688 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.683+0000 7f758b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c0a5fb0 0x7f758c0aa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:26.688 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.683+0000 7f758bfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f758c0a5be0 0x7f758c151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:26.688 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.683+0000 7f758b7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c0a5fb0 0x7f758c0aa940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54204/0 (socket says 172.21.15.40:54204) 2023-09-30T00:47:26.689 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.683+0000 7f758b7fe700 1 -- 172.21.15.40:0/3523941743 learned_addr learned my addr 172.21.15.40:0/3523941743 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:26.689 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.683+0000 7f758b7fe700 10 monclient(hunting): get_auth_request con 0x7f758c0a5fb0 auth_method 0 2023-09-30T00:47:26.689 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.683+0000 7f758b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:26.689 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.683+0000 7f758b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:26.689 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.683+0000 7f758b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:26.689 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.683+0000 7f758bfff700 10 monclient(hunting): get_auth_request con 0x7f758c0a5be0 auth_method 0 2023-09-30T00:47:26.689 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.683+0000 7f758bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:26.689 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.683+0000 7f758bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:26.689 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.683+0000 7f758bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:26.690 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.684+0000 7f758b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:26.690 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.684+0000 7f758b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:26.690 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.684+0000 7f758b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:26.690 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.684+0000 7f758bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:26.690 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.684+0000 7f758bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:26.690 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.684+0000 7f758bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:26.691 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.690+0000 7f758b7fe700 10 monclient(hunting): handle_auth_done global_id 4209 payload 824 2023-09-30T00:47:26.691 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.690+0000 7f758b7fe700 1 -- 172.21.15.40:0/3523941743 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f758c0a5be0 msgr2=0x7f758c151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:26.691 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.690+0000 7f758b7fe700 1 --2- 172.21.15.40:0/3523941743 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f758c0a5be0 0x7f758c151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:26.691 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.690+0000 7f758b7fe700 1 -- 172.21.15.40:0/3523941743 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f758c0a63e0 msgr2=0x7f758c0ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:26.691 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.690+0000 7f758b7fe700 1 --2- 172.21.15.40:0/3523941743 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f758c0a63e0 0x7f758c0ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:26.691 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.690+0000 7f758b7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:47:26.691 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.690+0000 7f758b7fe700 1 monclient: found mon.noname-a 2023-09-30T00:47:26.691 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.690+0000 7f758b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:26.691 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.690+0000 7f758b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:26.691 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.690+0000 7f758b7fe700 1 -- 172.21.15.40:0/3523941743 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7584009a20 con 0x7f758c0a5fb0 2023-09-30T00:47:26.693 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.692+0000 7f758b7fe700 1 --2- 172.21.15.40:0/3523941743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c0a5fb0 0x7f758c0aa940 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f758000ec80 tx=0x7f75800091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:26.694 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.693+0000 7f7573fff700 10 client.?.objecter ms_handle_connect 0x7f758c0a5fb0 2023-09-30T00:47:26.694 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.693+0000 7f7573fff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:26.698 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.697+0000 7f7573fff700 1 -- 172.21.15.40:0/3523941743 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758000c850 con 0x7f758c0a5fb0 2023-09-30T00:47:26.699 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f7573fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:26.699 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f7573fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:26.699 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f7573fff700 10 monclient: dump: 2023-09-30T00:47:26.699 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:26.699 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:26.699 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:26.699 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:26.699 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:26.699 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:26.699 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:26.700 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:26.700 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:26.700 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:26.700 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f7573fff700 10 monclient: _finish_auth 0 2023-09-30T00:47:26.700 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f7573fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:26.700 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f7573fff700 1 -- 172.21.15.40:0/3523941743 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758000c9b0 con 0x7f758c0a5fb0 2023-09-30T00:47:26.700 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f7573fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:26.700 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f7573fff700 1 -- 172.21.15.40:0/3523941743 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758000cd50 con 0x7f758c0a5fb0 2023-09-30T00:47:26.700 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f7573fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:26.700 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f7573fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:26.700 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f7573fff700 10 monclient: dump: 2023-09-30T00:47:26.700 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:26.700 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:26.701 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:26.701 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:26.701 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:26.701 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:26.701 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:26.701 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:26.701 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:26.701 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:26.701 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f758a7fc700 4 set_mon_vals no callback set 2023-09-30T00:47:26.701 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f75927a2700 5 monclient: authenticate success, global_id 4209 2023-09-30T00:47:26.701 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f75927a2700 10 monclient: _renew_subs 2023-09-30T00:47:26.701 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f75927a2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:26.701 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f75927a2700 1 -- 172.21.15.40:0/3523941743 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758c152720 con 0x7f758c0a5fb0 2023-09-30T00:47:26.702 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f75927a2700 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:26.702 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f75927a2700 10 monclient: _renew_subs 2023-09-30T00:47:26.702 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f75927a2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:26.702 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f75927a2700 1 -- 172.21.15.40:0/3523941743 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f758c0a8250 con 0x7f758c0a5fb0 2023-09-30T00:47:26.702 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.698+0000 7f75927a2700 1 librados: init done 2023-09-30T00:47:26.705 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.700+0000 7f75927a2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:26.705 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.700+0000 7f75927a2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:26.705 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.700+0000 7f75927a2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:26.705 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.700+0000 7f75927a2700 1 -- 172.21.15.40:0/3523941743 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758c0ab3a0 con 0x7f758c0a5fb0 2023-09-30T00:47:26.743 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.741+0000 7f7573fff700 1 -- 172.21.15.40:0/3523941743 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f7580013680 con 0x7f758c0a5fb0 2023-09-30T00:47:26.743 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.741+0000 7f7573fff700 1 --2- 172.21.15.40:0/3523941743 >> v2:172.21.15.135:6800/116426 conn(0x7f7578068880 0x7f757806ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:26.743 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.742+0000 7f7573fff700 1 -- 172.21.15.40:0/3523941743 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f7580083e90 con 0x7f758c0a5fb0 2023-09-30T00:47:26.743 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.742+0000 7f7573fff700 10 client.4209.objecter ms_dispatch 0x7f758c001550 osd_map(10..10 src has 1..10) v4 2023-09-30T00:47:26.743 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.742+0000 7f7573fff700 3 client.4209.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T00:47:26.743 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.742+0000 7f7573fff700 3 client.4209.objecter handle_osd_map decoding full epoch 10 2023-09-30T00:47:26.743 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.742+0000 7f7573fff700 20 client.4209.objecter dump_active .. 0 homeless 2023-09-30T00:47:26.743 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.742+0000 7f758bfff700 1 --2- 172.21.15.40:0/3523941743 >> v2:172.21.15.135:6800/116426 conn(0x7f7578068880 0x7f757806ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:26.743 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.742+0000 7f758bfff700 10 monclient: get_auth_request con 0x7f7578068880 auth_method 0 2023-09-30T00:47:26.744 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.742+0000 7f7573fff700 10 client.4209.objecter ms_handle_connect 0x7f7578068880 2023-09-30T00:47:26.744 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:26.742+0000 7f758bfff700 1 --2- 172.21.15.40:0/3523941743 >> v2:172.21.15.135:6800/116426 conn(0x7f7578068880 0x7f757806ad30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f757c007130 tx=0x7f757c007eb0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:27.202 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.202+0000 7f7573fff700 1 -- 172.21.15.40:0/3523941743 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f75800521a0 con 0x7f758c0a5fb0 2023-09-30T00:47:27.203 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.202+0000 7f7573fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:27.203 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.202+0000 7f7573fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:27.510 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.509+0000 7f75927a2700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:27.510 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.509+0000 7f75927a2700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:27.511 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.509+0000 7f75927a2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:27.511 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.509+0000 7f75927a2700 1 -- 172.21.15.40:0/3523941743 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f758c064a00 con 0x7f758c0a5fb0 2023-09-30T00:47:27.542 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.541+0000 7f7573fff700 1 -- 172.21.15.40:0/3523941743 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f7580017090 con 0x7f758c0a5fb0 2023-09-30T00:47:27.542 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.541+0000 7f7573fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:27.542 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.541+0000 7f7573fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:27.543 INFO:teuthology.misc.health.smithi040.stdout: 2023-09-30T00:47:27.543 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":10,"fsid":"238dba7c-9190-48bd-bb67-322ebc9ecbd0","created":"2023-09-30T00:46:45.822318+0000","modified":"2023-09-30T00:46:52.657791+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:46:52.657791+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"66e1da65-310e-4463-8aa0-431896832e64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d6dd269f-2eda-4ef7-9da1-2ed5b16291b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f0de27d-3674-4ff4-8451-4fcc90786c98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d18ec2f5-7e8f-4732-a523-adc495115cbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e2465fe-bb27-47bd-8ddf-f334cac6041a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7ea449db-9b22-48b8-a61b-1937eeeb6c8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c79b4829-6393-4f9a-a9f8-d2aacb505ae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:47:27.544 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.543+0000 7f75927a2700 10 librados: watch_flush enter 2023-09-30T00:47:27.544 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.543+0000 7f75927a2700 10 librados: watch_flush exit 2023-09-30T00:47:27.544 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.543+0000 7f75927a2700 20 client.4209.objecter shutdown clearing up homeless session... 2023-09-30T00:47:27.544 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.544+0000 7f75927a2700 10 client.4209.objecter successfully canceled tick 2023-09-30T00:47:27.545 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 1 -- 172.21.15.40:0/3523941743 >> v2:172.21.15.135:6800/116426 conn(0x7f7578068880 msgr2=0x7f757806ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:27.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 1 --2- 172.21.15.40:0/3523941743 >> v2:172.21.15.135:6800/116426 conn(0x7f7578068880 0x7f757806ad30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f757c007130 tx=0x7f757c007eb0 comp rx=0 tx=0).stop 2023-09-30T00:47:27.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 10 monclient: shutdown 2023-09-30T00:47:27.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:27.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 1 -- 172.21.15.40:0/3523941743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c0a5fb0 msgr2=0x7f758c0aa940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:27.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 1 --2- 172.21.15.40:0/3523941743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c0a5fb0 0x7f758c0aa940 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f758000ec80 tx=0x7f75800091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:27.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 1 -- 172.21.15.40:0/3523941743 shutdown_connections 2023-09-30T00:47:27.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 1 --2- 172.21.15.40:0/3523941743 >> v2:172.21.15.135:6800/116426 conn(0x7f7578068880 0x7f757806ad30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:27.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 1 --2- 172.21.15.40:0/3523941743 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f758c0a63e0 0x7f758c0ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:27.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 1 --2- 172.21.15.40:0/3523941743 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f758c0a5be0 0x7f758c151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:27.547 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 1 --2- 172.21.15.40:0/3523941743 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c0a5fb0 0x7f758c0aa940 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:27.547 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 1 -- 172.21.15.40:0/3523941743 >> 172.21.15.40:0/3523941743 conn(0x7f758c063280 msgr2=0x7f758c0b3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:27.547 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 1 -- 172.21.15.40:0/3523941743 shutdown_connections 2023-09-30T00:47:27.547 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 1 -- 172.21.15.40:0/3523941743 wait complete. 2023-09-30T00:47:27.547 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:27.545+0000 7f75927a2700 1 librados: shutdown 2023-09-30T00:47:27.559 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T00:47:27.856 INFO:tasks.ceph.osd.1.smithi040.stderr:==00:00:00:34.963 117237== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:28.002 INFO:tasks.ceph.osd.0.smithi040.stderr:==00:00:00:35.103 117238== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:28.041 INFO:tasks.ceph.osd.3.smithi040.stderr:==00:00:00:35.155 117236== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:28.501 INFO:tasks.ceph.osd.2.smithi040.stderr:==00:00:00:35.601 117239== Warning: unimplemented fcntl command: 1038 2023-09-30T00:47:33.453 INFO:tasks.ceph.osd.6.smithi135.stderr:2023-09-30T00:47:33.449+0000 40451c0 -1 osd.6 0 log_to_monitors true 2023-09-30T00:47:33.828 INFO:tasks.ceph.osd.4.smithi135.stderr:2023-09-30T00:47:33.825+0000 40451c0 -1 osd.4 0 log_to_monitors true 2023-09-30T00:47:34.089 INFO:tasks.ceph.osd.7.smithi135.stderr:2023-09-30T00:47:34.085+0000 40451c0 -1 osd.7 0 log_to_monitors true 2023-09-30T00:47:34.366 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:47:34.529 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.528+0000 7fbff1831700 10 monclient: get_monmap_and_config 2023-09-30T00:47:34.529 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.528+0000 7fbff1831700 10 monclient: build_initial_monmap 2023-09-30T00:47:34.529 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.528+0000 7fbff1831700 10 monclient: monmap: 2023-09-30T00:47:34.529 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:47:34.529 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:34.529 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:47:34.529151+0000 2023-09-30T00:47:34.529 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:47:34.529151+0000 2023-09-30T00:47:34.529 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:34.529 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:34.529 INFO:teuthology.misc.health.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:34.530 INFO:teuthology.misc.health.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:34.530 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:34.530 INFO:teuthology.misc.health.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:34.530 INFO:teuthology.misc.health.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:34.530 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:34.531 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.528+0000 7fbff1831700 1 Processor -- start 2023-09-30T00:47:34.531 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.529+0000 7fbff1831700 1 -- start start 2023-09-30T00:47:34.531 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.529+0000 7fbff1831700 10 monclient: init 2023-09-30T00:47:34.531 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbff1831700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:34.531 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbff1831700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-09-30T00:47:34.531 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbff1831700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbfec0a5be0 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:34.532 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbff1831700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbfec0a5fb0 0x7fbfec141d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:34.532 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbff1831700 10 monclient(hunting): picked mon.noname-c con 0x7fbfec0a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:34.532 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbff1831700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbfec0a63e0 0x7fbfec142240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:34.532 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbff1831700 10 monclient(hunting): picked mon.noname-a con 0x7fbfec0a63e0 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:34.532 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbff1831700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfec0a7750 con 0x7fbfec0a5be0 2023-09-30T00:47:34.532 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbff1831700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:34.532 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbff1831700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfec0a59d0 con 0x7fbfec0a63e0 2023-09-30T00:47:34.532 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbff1831700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:34.532 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbff1831700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfec0a56f0 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.532 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbff1831700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:34.532 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbff1831700 10 monclient(hunting): authenticate will time out at 5315.831196s 2023-09-30T00:47:34.532 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbfeaffd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbfec0a5be0 0x7fbfec13eba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42432/0 (socket says 172.21.15.40:42432) 2023-09-30T00:47:34.533 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.530+0000 7fbfeaffd700 1 -- 172.21.15.40:0/4024071442 learned_addr learned my addr 172.21.15.40:0/4024071442 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:34.533 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.531+0000 7fbfea7fc700 1 --2- 172.21.15.40:0/4024071442 >> v2:172.21.15.135:3300/0 conn(0x7fbfec0a5fb0 0x7fbfec141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:34.533 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.531+0000 7fbfea7fc700 10 monclient(hunting): get_auth_request con 0x7fbfec0a5fb0 auth_method 0 2023-09-30T00:47:34.533 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.531+0000 7fbfea7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:34.533 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.531+0000 7fbfea7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:34.533 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.531+0000 7fbfea7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:34.534 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.532+0000 7fbfea7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:34.534 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.532+0000 7fbfea7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:34.534 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.532+0000 7fbfea7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:34.536 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.535+0000 7fbfea7fc700 10 monclient(hunting): handle_auth_done global_id 4240 payload 274 2023-09-30T00:47:34.536 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.535+0000 7fbfea7fc700 1 -- 172.21.15.40:0/4024071442 >> v2:172.21.15.40:3300/0 conn(0x7fbfec0a63e0 msgr2=0x7fbfec142240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:34.536 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.535+0000 7fbfea7fc700 1 --2- 172.21.15.40:0/4024071442 >> v2:172.21.15.40:3300/0 conn(0x7fbfec0a63e0 0x7fbfec142240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:34.536 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.535+0000 7fbfea7fc700 1 -- 172.21.15.40:0/4024071442 >> v1:172.21.15.40:6789/0 conn(0x7fbfec0a5be0 legacy=0x7fbfec13eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:34.536 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.535+0000 7fbfea7fc700 10 monclient: _finish_hunting 0 2023-09-30T00:47:34.536 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.535+0000 7fbfea7fc700 1 monclient: found mon.noname-c 2023-09-30T00:47:34.536 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.535+0000 7fbfea7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:34.537 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.535+0000 7fbfea7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:47:34.537 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.535+0000 7fbfea7fc700 1 -- 172.21.15.40:0/4024071442 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec13f2b0 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.538 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.538+0000 7fbfea7fc700 1 --2- 172.21.15.40:0/4024071442 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbfec0a5fb0 0x7fbfec141d00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbfec0b2190 tx=0x7fbfe00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7947dfc1d648b8ba server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:34.545 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbfe9ffb700 1 -- 172.21.15.40:0/4024071442 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe000b6b0 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.545 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbfe9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:34.545 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbfe9ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:34.545 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbfe9ffb700 10 monclient: dump: 2023-09-30T00:47:34.545 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:34.545 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:34.545 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:34.545 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:34.545 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:34.545 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:34.545 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:34.546 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:34.546 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:34.546 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:34.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbfe9ffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:34.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbfe9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:34.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbfe9ffb700 1 -- 172.21.15.40:0/4024071442 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe000b810 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbfe9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:34.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbfe9ffb700 1 -- 172.21.15.40:0/4024071442 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe000bbb0 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbfe9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:34.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbfe9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:34.546 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbfe9ffb700 10 monclient: dump: 2023-09-30T00:47:34.546 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:34.547 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:34.547 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:34.547 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:34.547 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:34.547 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:34.547 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:34.547 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:34.547 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:34.547 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:34.547 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbff1831700 5 monclient: authenticate success, global_id 4240 2023-09-30T00:47:34.547 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbff1831700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:34.547 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.544+0000 7fbff1831700 4 set_mon_vals no callback set 2023-09-30T00:47:34.548 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.545+0000 7fbff1831700 10 monclient: shutdown 2023-09-30T00:47:34.548 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.545+0000 7fbff1831700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:34.548 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.545+0000 7fbff1831700 1 -- 172.21.15.40:0/4024071442 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbfec0a5fb0 msgr2=0x7fbfec141d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:34.549 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.545+0000 7fbff1831700 1 --2- 172.21.15.40:0/4024071442 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbfec0a5fb0 0x7fbfec141d00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbfec0b2190 tx=0x7fbfe00091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:34.549 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.545+0000 7fbff1831700 1 -- 172.21.15.40:0/4024071442 shutdown_connections 2023-09-30T00:47:34.549 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.545+0000 7fbff1831700 1 --2- 172.21.15.40:0/4024071442 >> v2:172.21.15.40:3300/0 conn(0x7fbfec0a63e0 0x7fbfec142240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:34.549 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.545+0000 7fbff1831700 1 --2- 172.21.15.40:0/4024071442 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbfec0a5fb0 0x7fbfec141d00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:34.549 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.545+0000 7fbff1831700 1 -- 172.21.15.40:0/4024071442 >> 172.21.15.40:0/4024071442 conn(0x7fbfec063280 msgr2=0x7fbfec063680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:34.549 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.545+0000 7fbff1831700 1 -- 172.21.15.40:0/4024071442 shutdown_connections 2023-09-30T00:47:34.549 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.545+0000 7fbff1831700 1 -- 172.21.15.40:0/4024071442 wait complete. 2023-09-30T00:47:34.549 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.546+0000 7fbff1831700 10 monclient: build_initial_monmap 2023-09-30T00:47:34.549 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.546+0000 7fbff1831700 10 monclient: monmap: 2023-09-30T00:47:34.549 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:47:34.549 INFO:teuthology.misc.health.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:34.549 INFO:teuthology.misc.health.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:34.550 INFO:teuthology.misc.health.smithi040.stderr:created 0.000000 2023-09-30T00:47:34.550 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:34.550 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:34.550 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:34.550 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:34.550 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:34.550 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:34.550 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.547+0000 7fbff1831700 1 librados: starting msgr at 2023-09-30T00:47:34.550 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.547+0000 7fbff1831700 1 librados: starting objecter 2023-09-30T00:47:34.550 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.547+0000 7fbff1831700 1 Processor -- start 2023-09-30T00:47:34.552 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.547+0000 7fbff1831700 1 -- start start 2023-09-30T00:47:34.552 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.547+0000 7fbff1831700 1 librados: setting wanted keys 2023-09-30T00:47:34.552 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.547+0000 7fbff1831700 1 librados: calling monclient init 2023-09-30T00:47:34.552 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.547+0000 7fbff1831700 10 monclient: init 2023-09-30T00:47:34.552 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:34.552 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:47:34.552 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfec0a5be0 0x7fbfec06dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:34.552 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 10 monclient(hunting): picked mon.noname-c con 0x7fbfec0a5be0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:34.552 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfec0a5fb0 0x7fbfec06ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:34.552 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 10 monclient(hunting): picked mon.noname-a con 0x7fbfec0a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:34.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbfec0a63e0 0x7fbfec137100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:34.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 10 monclient(hunting): picked mon.noname-b con 0x7fbfec0a63e0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:34.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:34.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec06bf90 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:34.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec136a00 con 0x7fbfec0a63e0 2023-09-30T00:47:34.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:34.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec06e450 con 0x7fbfec0a5be0 2023-09-30T00:47:34.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:34.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.548+0000 7fbff1831700 10 monclient(hunting): authenticate will time out at 5315.849549s 2023-09-30T00:47:34.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.549+0000 7fbfeb7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbfec0a63e0 0x7fbfec137100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:34.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.549+0000 7fbfeb7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbfec0a63e0 0x7fbfec137100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:60656/0 (socket says 172.21.15.40:60656) 2023-09-30T00:47:34.553 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.549+0000 7fbfeb7fe700 1 -- 172.21.15.40:0/288110988 learned_addr learned my addr 172.21.15.40:0/288110988 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:34.554 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.549+0000 7fbfeb7fe700 10 monclient(hunting): get_auth_request con 0x7fbfec0a63e0 auth_method 0 2023-09-30T00:47:34.554 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.549+0000 7fbfeb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:34.554 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.549+0000 7fbfeb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:34.554 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.549+0000 7fbfeb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:34.554 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.549+0000 7fbfea7fc700 1 --2- 172.21.15.40:0/288110988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfec0a5fb0 0x7fbfec06ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:34.554 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.550+0000 7fbfea7fc700 10 monclient(hunting): get_auth_request con 0x7fbfec0a5fb0 auth_method 0 2023-09-30T00:47:34.554 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.550+0000 7fbfea7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:34.554 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.550+0000 7fbfea7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:34.554 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.550+0000 7fbfea7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:34.554 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.550+0000 7fbfeb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:34.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.550+0000 7fbfeb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:34.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.550+0000 7fbfeb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:34.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.550+0000 7fbfeaffd700 1 --2- 172.21.15.40:0/288110988 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfec0a5be0 0x7fbfec06dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:34.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.551+0000 7fbfeaffd700 10 monclient(hunting): get_auth_request con 0x7fbfec0a5be0 auth_method 0 2023-09-30T00:47:34.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.551+0000 7fbfeaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:34.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.551+0000 7fbfeaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:34.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.551+0000 7fbfeaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:34.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.551+0000 7fbfea7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:34.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.551+0000 7fbfea7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:34.555 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.551+0000 7fbfea7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:34.556 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.551+0000 7fbfeaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:34.556 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.551+0000 7fbfeaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:34.556 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.551+0000 7fbfeaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:34.556 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.556+0000 7fbfea7fc700 10 monclient(hunting): handle_auth_done global_id 4215 payload 824 2023-09-30T00:47:34.557 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.556+0000 7fbfea7fc700 1 -- 172.21.15.40:0/288110988 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfec0a5be0 msgr2=0x7fbfec06dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:34.557 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.556+0000 7fbfea7fc700 1 --2- 172.21.15.40:0/288110988 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfec0a5be0 0x7fbfec06dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:34.557 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.556+0000 7fbfea7fc700 1 -- 172.21.15.40:0/288110988 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbfec0a63e0 msgr2=0x7fbfec137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:34.557 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.556+0000 7fbfea7fc700 1 --2- 172.21.15.40:0/288110988 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbfec0a63e0 0x7fbfec137100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:34.557 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.556+0000 7fbfea7fc700 10 monclient: _finish_hunting 0 2023-09-30T00:47:34.557 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.556+0000 7fbfea7fc700 1 monclient: found mon.noname-a 2023-09-30T00:47:34.557 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.556+0000 7fbfea7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:34.557 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.556+0000 7fbfea7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:34.557 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.556+0000 7fbfea7fc700 1 -- 172.21.15.40:0/288110988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe0009420 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.561 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.560+0000 7fbfea7fc700 1 --2- 172.21.15.40:0/288110988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfec0a5fb0 0x7fbfec06ba50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0016000 tx=0x7fbfe0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:34.561 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.560+0000 7fbfd2ffd700 10 client.?.objecter ms_handle_connect 0x7fbfec0a5fb0 2023-09-30T00:47:34.561 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.560+0000 7fbfd2ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:34.561 INFO:tasks.ceph.osd.5.smithi135.stderr:2023-09-30T00:47:34.558+0000 40451c0 -1 osd.5 0 log_to_monitors true 2023-09-30T00:47:34.566 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbfd2ffd700 1 -- 172.21.15.40:0/288110988 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0000c40 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.567 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbfd2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:34.567 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbfd2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:34.567 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbfd2ffd700 10 monclient: dump: 2023-09-30T00:47:34.567 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:34.567 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:34.567 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:34.567 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:34.567 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:34.567 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:34.567 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:34.567 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:34.568 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:34.568 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:34.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbfd2ffd700 10 monclient: _finish_auth 0 2023-09-30T00:47:34.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbfd2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:34.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbff1831700 5 monclient: authenticate success, global_id 4215 2023-09-30T00:47:34.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbff1831700 10 monclient: _renew_subs 2023-09-30T00:47:34.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbff1831700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:34.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbff1831700 1 -- 172.21.15.40:0/288110988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfec06c460 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbff1831700 10 client.4215.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:34.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbff1831700 10 monclient: _renew_subs 2023-09-30T00:47:34.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbff1831700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:34.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbff1831700 1 -- 172.21.15.40:0/288110988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfec137800 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.566+0000 7fbff1831700 1 librados: init done 2023-09-30T00:47:34.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.567+0000 7fbfd2ffd700 1 -- 172.21.15.40:0/288110988 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe000bb90 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.567+0000 7fbfd2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:34.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.567+0000 7fbfd2ffd700 1 -- 172.21.15.40:0/288110988 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe002f440 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.567+0000 7fbfd2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:34.570 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.567+0000 7fbfd2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:34.570 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.567+0000 7fbfd2ffd700 10 monclient: dump: 2023-09-30T00:47:34.570 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:34.570 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:34.570 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:34.570 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:34.570 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:34.570 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:34.570 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:34.570 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:34.570 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:34.571 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:34.571 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.567+0000 7fbfe97fa700 4 set_mon_vals no callback set 2023-09-30T00:47:34.571 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.568+0000 7fbff1831700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:34.571 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.568+0000 7fbff1831700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:34.571 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.568+0000 7fbff1831700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:34.571 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.568+0000 7fbff1831700 1 -- 172.21.15.40:0/288110988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfcc0050b0 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.610 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.609+0000 7fbfd2ffd700 1 -- 172.21.15.40:0/288110988 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7fbfe0023030 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.611 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.610+0000 7fbfd2ffd700 1 --2- 172.21.15.40:0/288110988 >> v2:172.21.15.135:6800/116426 conn(0x7fbfd8068800 0x7fbfd806acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:34.611 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.610+0000 7fbfd2ffd700 1 -- 172.21.15.40:0/288110988 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fbfe0085180 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.611 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.610+0000 7fbfd2ffd700 10 client.4215.objecter ms_dispatch 0x7fbfec001550 osd_map(10..10 src has 1..10) v4 2023-09-30T00:47:34.611 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.610+0000 7fbfd2ffd700 3 client.4215.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T00:47:34.611 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.610+0000 7fbfd2ffd700 3 client.4215.objecter handle_osd_map decoding full epoch 10 2023-09-30T00:47:34.611 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.610+0000 7fbfd2ffd700 20 client.4215.objecter dump_active .. 0 homeless 2023-09-30T00:47:34.611 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.610+0000 7fbfeaffd700 1 --2- 172.21.15.40:0/288110988 >> v2:172.21.15.135:6800/116426 conn(0x7fbfd8068800 0x7fbfd806acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:34.612 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.610+0000 7fbfeaffd700 10 monclient: get_auth_request con 0x7fbfd8068800 auth_method 0 2023-09-30T00:47:34.612 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.611+0000 7fbfd2ffd700 10 client.4215.objecter ms_handle_connect 0x7fbfd8068800 2023-09-30T00:47:34.612 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.611+0000 7fbfeaffd700 1 --2- 172.21.15.40:0/288110988 >> v2:172.21.15.135:6800/116426 conn(0x7fbfd8068800 0x7fbfd806acb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc000c00 tx=0x7fbfdc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:34.976 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.975+0000 7fbfd2ffd700 1 -- 172.21.15.40:0/288110988 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbfe0053510 con 0x7fbfec0a5fb0 2023-09-30T00:47:34.976 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.975+0000 7fbfd2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:34.976 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:34.975+0000 7fbfd2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:35.352 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.351+0000 7fbff1831700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:35.352 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.352+0000 7fbff1831700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:35.353 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.352+0000 7fbff1831700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:35.353 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.352+0000 7fbff1831700 1 -- 172.21.15.40:0/288110988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfcc005630 con 0x7fbfec0a5fb0 2023-09-30T00:47:35.594 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.593+0000 7fbfd2ffd700 1 -- 172.21.15.40:0/288110988 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fbfe0056b30 con 0x7fbfec0a5fb0 2023-09-30T00:47:35.594 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.593+0000 7fbfd2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:35.594 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.593+0000 7fbfd2ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:35.595 INFO:teuthology.misc.health.smithi040.stdout: 2023-09-30T00:47:35.595 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":11,"fsid":"238dba7c-9190-48bd-bb67-322ebc9ecbd0","created":"2023-09-30T00:46:45.822318+0000","modified":"2023-09-30T00:47:35.239248+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:46:52.657791+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"66e1da65-310e-4463-8aa0-431896832e64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d6dd269f-2eda-4ef7-9da1-2ed5b16291b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f0de27d-3674-4ff4-8451-4fcc90786c98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d18ec2f5-7e8f-4732-a523-adc495115cbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e2465fe-bb27-47bd-8ddf-f334cac6041a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7ea449db-9b22-48b8-a61b-1937eeeb6c8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c79b4829-6393-4f9a-a9f8-d2aacb505ae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:47:35.597 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.597+0000 7fbff1831700 10 librados: watch_flush enter 2023-09-30T00:47:35.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.598+0000 7fbff1831700 10 librados: watch_flush exit 2023-09-30T00:47:35.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.598+0000 7fbff1831700 20 client.4215.objecter shutdown clearing up homeless session... 2023-09-30T00:47:35.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.598+0000 7fbff1831700 10 client.4215.objecter successfully canceled tick 2023-09-30T00:47:35.599 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.598+0000 7fbff1831700 1 -- 172.21.15.40:0/288110988 >> v2:172.21.15.135:6800/116426 conn(0x7fbfd8068800 msgr2=0x7fbfd806acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:35.599 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.598+0000 7fbff1831700 1 --2- 172.21.15.40:0/288110988 >> v2:172.21.15.135:6800/116426 conn(0x7fbfd8068800 0x7fbfd806acb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc000c00 tx=0x7fbfdc009040 comp rx=0 tx=0).stop 2023-09-30T00:47:35.599 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.598+0000 7fbff1831700 10 monclient: shutdown 2023-09-30T00:47:35.599 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.598+0000 7fbff1831700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:35.599 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.598+0000 7fbff1831700 1 -- 172.21.15.40:0/288110988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfec0a5fb0 msgr2=0x7fbfec06ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:35.599 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.598+0000 7fbff1831700 1 --2- 172.21.15.40:0/288110988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfec0a5fb0 0x7fbfec06ba50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0016000 tx=0x7fbfe0017000 comp rx=0 tx=0).stop 2023-09-30T00:47:35.600 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.599+0000 7fbff1831700 1 -- 172.21.15.40:0/288110988 shutdown_connections 2023-09-30T00:47:35.600 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.599+0000 7fbff1831700 1 --2- 172.21.15.40:0/288110988 >> v2:172.21.15.135:6800/116426 conn(0x7fbfd8068800 0x7fbfd806acb0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:35.600 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.599+0000 7fbff1831700 1 --2- 172.21.15.40:0/288110988 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbfec0a63e0 0x7fbfec137100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:35.600 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.599+0000 7fbff1831700 1 --2- 172.21.15.40:0/288110988 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbfec0a5be0 0x7fbfec06dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:35.600 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.599+0000 7fbff1831700 1 --2- 172.21.15.40:0/288110988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbfec0a5fb0 0x7fbfec06ba50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:35.600 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.599+0000 7fbff1831700 1 -- 172.21.15.40:0/288110988 >> 172.21.15.40:0/288110988 conn(0x7fbfec063280 msgr2=0x7fbfec0b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:35.600 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.599+0000 7fbff1831700 1 -- 172.21.15.40:0/288110988 shutdown_connections 2023-09-30T00:47:35.600 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.599+0000 7fbff1831700 1 -- 172.21.15.40:0/288110988 wait complete. 2023-09-30T00:47:35.603 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:35.603+0000 7fbff1831700 1 librados: shutdown 2023-09-30T00:47:35.617 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T00:47:36.982 INFO:tasks.ceph.osd.6.smithi135.stderr:2023-09-30T00:47:36.978+0000 e689700 -1 osd.6 0 waiting for initial osdmap 2023-09-30T00:47:37.483 INFO:tasks.ceph.osd.6.smithi135.stderr:2023-09-30T00:47:37.469+0000 14890700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:47:38.121 INFO:tasks.ceph.osd.5.smithi135.stderr:2023-09-30T00:47:38.118+0000 fe8c700 -1 osd.5 0 waiting for initial osdmap 2023-09-30T00:47:38.142 INFO:tasks.ceph.osd.4.smithi135.stderr:2023-09-30T00:47:38.126+0000 fe8c700 -1 osd.4 0 waiting for initial osdmap 2023-09-30T00:47:38.186 INFO:tasks.ceph.osd.7.smithi135.stderr:2023-09-30T00:47:38.177+0000 e689700 -1 osd.7 0 waiting for initial osdmap 2023-09-30T00:47:38.685 INFO:tasks.ceph.osd.7.smithi135.stderr:2023-09-30T00:47:38.671+0000 14890700 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:47:38.693 INFO:tasks.ceph.osd.4.smithi135.stderr:2023-09-30T00:47:38.678+0000 14890700 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:47:38.719 INFO:tasks.ceph.osd.5.smithi135.stderr:2023-09-30T00:47:38.704+0000 14890700 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:47:38.795 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-09-30T00:47:38.791+0000 40451c0 -1 osd.1 0 log_to_monitors true 2023-09-30T00:47:39.210 INFO:tasks.ceph.osd.3.smithi040.stderr:2023-09-30T00:47:39.201+0000 40451c0 -1 osd.3 0 log_to_monitors true 2023-09-30T00:47:39.336 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-09-30T00:47:39.332+0000 40451c0 -1 osd.0 0 log_to_monitors true 2023-09-30T00:47:39.356 INFO:tasks.ceph.osd.2.smithi040.stderr:2023-09-30T00:47:39.352+0000 40451c0 -1 osd.2 0 log_to_monitors true 2023-09-30T00:47:42.423 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:47:42.559 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.559+0000 7f1bd38da700 10 monclient: get_monmap_and_config 2023-09-30T00:47:42.560 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.559+0000 7f1bd38da700 10 monclient: build_initial_monmap 2023-09-30T00:47:42.560 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.559+0000 7f1bd38da700 10 monclient: monmap: 2023-09-30T00:47:42.560 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:47:42.560 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:42.560 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:47:42.560310+0000 2023-09-30T00:47:42.560 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:47:42.560310+0000 2023-09-30T00:47:42.560 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:42.560 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:42.560 INFO:teuthology.misc.health.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:42.561 INFO:teuthology.misc.health.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:42.561 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:42.561 INFO:teuthology.misc.health.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:42.561 INFO:teuthology.misc.health.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:42.561 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:42.562 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.559+0000 7f1bd38da700 1 Processor -- start 2023-09-30T00:47:42.562 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.560+0000 7f1bd38da700 1 -- start start 2023-09-30T00:47:42.562 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.560+0000 7f1bd38da700 10 monclient: init 2023-09-30T00:47:42.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.560+0000 7f1bd38da700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:42.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.560+0000 7f1bd38da700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2023-09-30T00:47:42.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.560+0000 7f1bd38da700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1bcc0a5be0 0x7f1bcc13eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:42.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.560+0000 7f1bd38da700 10 monclient(hunting): picked mon.noname-a con 0x7f1bcc0a5be0 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:42.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.560+0000 7f1bd38da700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1bcc0a5fb0 0x7f1bcc141d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:42.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.561+0000 7f1bd38da700 10 monclient(hunting): picked mon.noname-c con 0x7f1bcc0a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:42.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.561+0000 7f1bd38da700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bcc13f0e0 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:42.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.561+0000 7f1bd38da700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc0a7750 con 0x7f1bcc13f0e0 2023-09-30T00:47:42.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.561+0000 7f1bd38da700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:42.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.561+0000 7f1bd38da700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bcc142950 con 0x7f1bcc0a5be0 2023-09-30T00:47:42.563 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.561+0000 7f1bd38da700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:42.564 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.561+0000 7f1bd38da700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bcc0a59d0 con 0x7f1bcc0a5fb0 2023-09-30T00:47:42.564 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.561+0000 7f1bd38da700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:42.564 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.561+0000 7f1bd38da700 10 monclient(hunting): authenticate will time out at 5323.862088s 2023-09-30T00:47:42.564 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-09-30T00:47:42.555+0000 e689700 -1 osd.1 0 waiting for initial osdmap 2023-09-30T00:47:42.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.561+0000 7f1bd0e75700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1bcc0a5fb0 0x7f1bcc141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:42.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.562+0000 7f1bd1e77700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f1bcc13f0e0 0x7f1bcc142240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:45698/0 (socket says 172.21.15.40:45698) 2023-09-30T00:47:42.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.562+0000 7f1bd1e77700 1 -- 172.21.15.40:0/3061994829 learned_addr learned my addr 172.21.15.40:0/3061994829 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:42.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.562+0000 7f1bd0e75700 10 monclient(hunting): get_auth_request con 0x7f1bcc0a5fb0 auth_method 0 2023-09-30T00:47:42.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.562+0000 7f1bd0e75700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:42.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.562+0000 7f1bd0e75700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:42.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.562+0000 7f1bd0e75700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:42.565 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.564+0000 7f1bd1676700 1 --2- 172.21.15.40:0/3061994829 >> v2:172.21.15.40:3300/0 conn(0x7f1bcc0a5be0 0x7f1bcc13eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:42.566 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.564+0000 7f1bd0e75700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:42.566 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.564+0000 7f1bd0e75700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:42.566 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.564+0000 7f1bd0e75700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:42.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.568+0000 7f1bd1676700 10 monclient(hunting): get_auth_request con 0x7f1bcc0a5be0 auth_method 0 2023-09-30T00:47:42.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.568+0000 7f1bd1676700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:42.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.568+0000 7f1bd1676700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:42.568 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.568+0000 7f1bd1676700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:42.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.568+0000 7f1bd0e75700 10 monclient(hunting): handle_auth_done global_id 4267 payload 274 2023-09-30T00:47:42.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.568+0000 7f1bd0e75700 1 -- 172.21.15.40:0/3061994829 >> v2:172.21.15.40:3300/0 conn(0x7f1bcc0a5be0 msgr2=0x7f1bcc13eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:42.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.568+0000 7f1bd0e75700 1 --2- 172.21.15.40:0/3061994829 >> v2:172.21.15.40:3300/0 conn(0x7f1bcc0a5be0 0x7f1bcc13eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:42.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.568+0000 7f1bd0e75700 1 -- 172.21.15.40:0/3061994829 >> v1:172.21.15.135:6789/0 conn(0x7f1bcc13f0e0 legacy=0x7f1bcc142240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:42.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.568+0000 7f1bd0e75700 10 monclient: _finish_hunting 0 2023-09-30T00:47:42.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.568+0000 7f1bd0e75700 1 monclient: found mon.noname-c 2023-09-30T00:47:42.569 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.568+0000 7f1bd0e75700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:42.570 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.568+0000 7f1bd0e75700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:47:42.570 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.568+0000 7f1bd0e75700 1 -- 172.21.15.40:0/3061994829 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc0a56f0 con 0x7f1bcc0a5fb0 2023-09-30T00:47:42.571 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.571+0000 7f1bd0e75700 1 --2- 172.21.15.40:0/3061994829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bcc0a5fb0 0x7f1bcc141d00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc0b2190 tx=0x7f1bbc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15e01757feb91f87 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:42.588 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bc3fff700 1 -- 172.21.15.40:0/3061994829 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc00b6b0 con 0x7f1bcc0a5fb0 2023-09-30T00:47:42.588 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bc3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:42.588 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bc3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:42.588 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bc3fff700 10 monclient: dump: 2023-09-30T00:47:42.588 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:42.588 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:42.588 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:42.588 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:42.588 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:42.589 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:42.589 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:42.589 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:42.589 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:42.589 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:42.589 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bc3fff700 10 monclient: _finish_auth 0 2023-09-30T00:47:42.589 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bc3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:42.589 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bc3fff700 1 -- 172.21.15.40:0/3061994829 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc00b810 con 0x7f1bcc0a5fb0 2023-09-30T00:47:42.589 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bd38da700 5 monclient: authenticate success, global_id 4267 2023-09-30T00:47:42.589 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bd38da700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:42.589 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bc3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:42.589 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bd38da700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:42.590 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bc3fff700 1 -- 172.21.15.40:0/3061994829 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc00bbb0 con 0x7f1bcc0a5fb0 2023-09-30T00:47:42.590 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bd38da700 4 set_mon_vals no callback set 2023-09-30T00:47:42.590 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bc3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:42.590 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bc3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:42.590 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.587+0000 7f1bc3fff700 10 monclient: dump: 2023-09-30T00:47:42.590 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:42.590 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:42.590 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:42.590 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:42.590 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:42.590 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:42.590 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:42.590 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:42.591 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:42.591 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:42.592 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.588+0000 7f1bd38da700 10 monclient: shutdown 2023-09-30T00:47:42.593 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.588+0000 7f1bd38da700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:42.593 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.588+0000 7f1bd38da700 1 -- 172.21.15.40:0/3061994829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bcc0a5fb0 msgr2=0x7f1bcc141d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:42.593 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.588+0000 7f1bd38da700 1 --2- 172.21.15.40:0/3061994829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bcc0a5fb0 0x7f1bcc141d00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc0b2190 tx=0x7f1bbc0091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:42.593 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.588+0000 7f1bd38da700 1 -- 172.21.15.40:0/3061994829 shutdown_connections 2023-09-30T00:47:42.593 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.588+0000 7f1bd38da700 1 --2- 172.21.15.40:0/3061994829 >> v2:172.21.15.40:3300/0 conn(0x7f1bcc0a5be0 0x7f1bcc13eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:42.593 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.588+0000 7f1bd38da700 1 --2- 172.21.15.40:0/3061994829 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bcc0a5fb0 0x7f1bcc141d00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:42.593 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.588+0000 7f1bd38da700 1 -- 172.21.15.40:0/3061994829 >> 172.21.15.40:0/3061994829 conn(0x7f1bcc063280 msgr2=0x7f1bcc063680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:42.593 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.589+0000 7f1bd38da700 1 -- 172.21.15.40:0/3061994829 shutdown_connections 2023-09-30T00:47:42.593 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.589+0000 7f1bd38da700 1 -- 172.21.15.40:0/3061994829 wait complete. 2023-09-30T00:47:42.593 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.590+0000 7f1bd38da700 10 monclient: build_initial_monmap 2023-09-30T00:47:42.593 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.590+0000 7f1bd38da700 10 monclient: monmap: 2023-09-30T00:47:42.593 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:47:42.594 INFO:teuthology.misc.health.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:42.594 INFO:teuthology.misc.health.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:42.594 INFO:teuthology.misc.health.smithi040.stderr:created 0.000000 2023-09-30T00:47:42.594 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:42.594 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:42.594 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:42.594 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:42.594 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:42.594 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:42.594 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.590+0000 7f1bd38da700 1 librados: starting msgr at 2023-09-30T00:47:42.594 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.590+0000 7f1bd38da700 1 librados: starting objecter 2023-09-30T00:47:42.594 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.590+0000 7f1bd38da700 1 Processor -- start 2023-09-30T00:47:42.595 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.590+0000 7f1bd38da700 1 -- start start 2023-09-30T00:47:42.595 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.590+0000 7f1bd38da700 1 librados: setting wanted keys 2023-09-30T00:47:42.595 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.590+0000 7f1bd38da700 1 librados: calling monclient init 2023-09-30T00:47:42.595 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.590+0000 7f1bd38da700 10 monclient: init 2023-09-30T00:47:42.595 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:42.595 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:47:42.595 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bcc0a5be0 0x7f1bcc151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:42.595 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 10 monclient(hunting): picked mon.noname-a con 0x7f1bcc0a5be0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:42.595 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bcc0a5fb0 0x7f1bcc0aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:42.595 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 10 monclient(hunting): picked mon.noname-c con 0x7f1bcc0a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:42.595 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bcc13f0e0 0x7f1bcc0aac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:42.595 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 10 monclient(hunting): picked mon.noname-b con 0x7f1bcc13f0e0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:42.596 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:42.596 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc0aa7f0 con 0x7f1bcc0a5be0 2023-09-30T00:47:42.596 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:42.596 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc0ac480 con 0x7f1bcc13f0e0 2023-09-30T00:47:42.596 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:42.596 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc0a7b90 con 0x7f1bcc0a5fb0 2023-09-30T00:47:42.596 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:42.596 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.592+0000 7f1bd38da700 10 monclient(hunting): authenticate will time out at 5323.893130s 2023-09-30T00:47:42.597 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.593+0000 7f1bd1676700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bcc0a5be0 0x7f1bcc151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:42.597 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.594+0000 7f1bd0e75700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bcc0a5fb0 0x7f1bcc0aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:42.597 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.595+0000 7f1bd0e75700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bcc0a5fb0 0x7f1bcc0aa090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:56946/0 (socket says 172.21.15.40:56946) 2023-09-30T00:47:42.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.595+0000 7f1bd0e75700 1 -- 172.21.15.40:0/3801083530 learned_addr learned my addr 172.21.15.40:0/3801083530 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:42.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.595+0000 7f1bd0e75700 10 monclient(hunting): get_auth_request con 0x7f1bcc0a5fb0 auth_method 0 2023-09-30T00:47:42.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.595+0000 7f1bd0e75700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:42.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.595+0000 7f1bd0e75700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:42.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.595+0000 7f1bd0e75700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:42.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.595+0000 7f1bd1676700 10 monclient(hunting): get_auth_request con 0x7f1bcc0a5be0 auth_method 0 2023-09-30T00:47:42.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.595+0000 7f1bd1676700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:42.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.595+0000 7f1bd1676700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:42.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.595+0000 7f1bd1676700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:42.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.595+0000 7f1bd1e77700 1 --2- 172.21.15.40:0/3801083530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bcc13f0e0 0x7f1bcc0aac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:42.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.596+0000 7f1bd1676700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:42.598 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.596+0000 7f1bd1676700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:42.599 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.596+0000 7f1bd1676700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:42.599 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.596+0000 7f1bd0e75700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:42.599 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.596+0000 7f1bd0e75700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:42.599 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.596+0000 7f1bd0e75700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:42.603 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.602+0000 7f1bd1676700 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2023-09-30T00:47:42.603 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.602+0000 7f1bd1676700 1 -- 172.21.15.40:0/3801083530 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bcc0a5fb0 msgr2=0x7f1bcc0aa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:42.603 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.602+0000 7f1bd1676700 1 --2- 172.21.15.40:0/3801083530 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bcc0a5fb0 0x7f1bcc0aa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:42.604 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.602+0000 7f1bd1676700 1 -- 172.21.15.40:0/3801083530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bcc13f0e0 msgr2=0x7f1bcc0aac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:42.604 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.602+0000 7f1bd1676700 1 --2- 172.21.15.40:0/3801083530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bcc13f0e0 0x7f1bcc0aac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:42.604 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.602+0000 7f1bd1676700 10 monclient: _finish_hunting 0 2023-09-30T00:47:42.604 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.602+0000 7f1bd1676700 1 monclient: found mon.noname-a 2023-09-30T00:47:42.604 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.602+0000 7f1bd1676700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:42.604 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.602+0000 7f1bd1676700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:42.604 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.602+0000 7f1bd1676700 1 -- 172.21.15.40:0/3801083530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bbc009420 con 0x7f1bcc0a5be0 2023-09-30T00:47:42.703 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.702+0000 7f1bd1676700 1 --2- 172.21.15.40:0/3801083530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bcc0a5be0 0x7f1bcc151420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1bc800d8c0 tx=0x7f1bc8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:42.703 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.702+0000 7f1bc17fa700 10 client.?.objecter ms_handle_connect 0x7f1bcc0a5be0 2023-09-30T00:47:42.703 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.702+0000 7f1bc17fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:42.719 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bc17fa700 1 -- 172.21.15.40:0/3801083530 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc80125f0 con 0x7f1bcc0a5be0 2023-09-30T00:47:42.719 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bc17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:42.719 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bc17fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:42.719 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bc17fa700 10 monclient: dump: 2023-09-30T00:47:42.719 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:42.719 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:42.719 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:42.720 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:42.720 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:42.720 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:42.720 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:42.720 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:42.720 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:42.720 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:42.720 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bc17fa700 10 monclient: _finish_auth 0 2023-09-30T00:47:42.720 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bc17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:42.720 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bd38da700 5 monclient: authenticate success, global_id 4239 2023-09-30T00:47:42.720 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bd38da700 10 monclient: _renew_subs 2023-09-30T00:47:42.720 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bd38da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:42.720 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bd38da700 1 -- 172.21.15.40:0/3801083530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bcc151e80 con 0x7f1bcc0a5be0 2023-09-30T00:47:42.721 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bd38da700 10 client.4239.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:42.721 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bd38da700 10 monclient: _renew_subs 2023-09-30T00:47:42.721 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bd38da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:42.721 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bd38da700 1 -- 172.21.15.40:0/3801083530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bcc0ab400 con 0x7f1bcc0a5be0 2023-09-30T00:47:42.721 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.718+0000 7f1bd38da700 1 librados: init done 2023-09-30T00:47:42.721 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.719+0000 7f1bd38da700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:42.722 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.719+0000 7f1bd38da700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:42.722 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.719+0000 7f1bd38da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:42.722 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.719+0000 7f1bd38da700 1 -- 172.21.15.40:0/3801083530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bcc064a00 con 0x7f1bcc0a5be0 2023-09-30T00:47:42.732 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.732+0000 7f1bc17fa700 1 -- 172.21.15.40:0/3801083530 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc80073c0 con 0x7f1bcc0a5be0 2023-09-30T00:47:42.732 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.732+0000 7f1bc17fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:42.732 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.732+0000 7f1bc17fa700 1 -- 172.21.15.40:0/3801083530 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc8012960 con 0x7f1bcc0a5be0 2023-09-30T00:47:42.733 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.732+0000 7f1bc17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:42.733 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.732+0000 7f1bc17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:42.733 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.732+0000 7f1bc17fa700 10 monclient: dump: 2023-09-30T00:47:42.733 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:42.733 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:42.733 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:42.733 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:42.733 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:42.733 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:42.734 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:42.734 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:42.734 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:42.734 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:42.734 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:42.732+0000 7f1bc37fe700 4 set_mon_vals no callback set 2023-09-30T00:47:43.056 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.056+0000 7f1bc17fa700 1 -- 172.21.15.40:0/3801083530 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f1bc8012ac0 con 0x7f1bcc0a5be0 2023-09-30T00:47:43.058 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.056+0000 7f1bc17fa700 1 --2- 172.21.15.40:0/3801083530 >> v2:172.21.15.135:6800/116426 conn(0x7f1bb0068800 0x7f1bb006acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:43.058 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.056+0000 7f1bc17fa700 1 -- 172.21.15.40:0/3801083530 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3125+0+0 (secure 0 0 0) 0x7f1bc8084800 con 0x7f1bcc0a5be0 2023-09-30T00:47:43.058 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.056+0000 7f1bc17fa700 10 client.4239.objecter ms_dispatch 0x7f1bcc001550 osd_map(17..17 src has 1..17) v4 2023-09-30T00:47:43.058 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.056+0000 7f1bc17fa700 3 client.4239.objecter handle_osd_map got epochs [17,17] > 0 2023-09-30T00:47:43.058 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.056+0000 7f1bc17fa700 3 client.4239.objecter handle_osd_map decoding full epoch 17 2023-09-30T00:47:43.058 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.057+0000 7f1bc17fa700 20 client.4239.objecter dump_active .. 0 homeless 2023-09-30T00:47:43.058 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.057+0000 7f1bc17fa700 1 -- 172.21.15.40:0/3801083530 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1bc80af640 con 0x7f1bcc0a5be0 2023-09-30T00:47:43.058 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.057+0000 7f1bc17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:43.058 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.057+0000 7f1bc17fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:43.059 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.057+0000 7f1bd0e75700 1 --2- 172.21.15.40:0/3801083530 >> v2:172.21.15.135:6800/116426 conn(0x7f1bb0068800 0x7f1bb006acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:43.059 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.057+0000 7f1bd0e75700 10 monclient: get_auth_request con 0x7f1bb0068800 auth_method 0 2023-09-30T00:47:43.059 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.057+0000 7f1bc17fa700 10 client.4239.objecter ms_handle_connect 0x7f1bb0068800 2023-09-30T00:47:43.059 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.057+0000 7f1bd0e75700 1 --2- 172.21.15.40:0/3801083530 >> v2:172.21.15.135:6800/116426 conn(0x7f1bb0068800 0x7f1bb006acb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009270 tx=0x7f1bbc017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:43.309 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.308+0000 7f1bd38da700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:43.309 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.308+0000 7f1bd38da700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:43.309 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.308+0000 7f1bd38da700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:43.309 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.308+0000 7f1bd38da700 1 -- 172.21.15.40:0/3801083530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bcc0ac5c0 con 0x7f1bcc0a5be0 2023-09-30T00:47:43.314 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-09-30T00:47:43.297+0000 14890700 -1 osd.1 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:47:43.414 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.414+0000 7f1bc17fa700 1 -- 172.21.15.40:0/3801083530 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+9528 (secure 0 0 0) 0x7f1bcc0ac5c0 con 0x7f1bcc0a5be0 2023-09-30T00:47:43.414 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.414+0000 7f1bc17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:43.415 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.414+0000 7f1bc17fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:43.417 INFO:teuthology.misc.health.smithi040.stdout: 2023-09-30T00:47:43.418 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":18,"fsid":"238dba7c-9190-48bd-bb67-322ebc9ecbd0","created":"2023-09-30T00:46:45.822318+0000","modified":"2023-09-30T00:47:43.145213+0000","last_up_change":"2023-09-30T00:47:39.834296+0000","last_in_change":"2023-09-30T00:46:52.657791+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:47:40.523068+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66e1da65-310e-4463-8aa0-431896832e64","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d6dd269f-2eda-4ef7-9da1-2ed5b16291b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8f0de27d-3674-4ff4-8451-4fcc90786c98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d18ec2f5-7e8f-4732-a523-adc495115cbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4e2465fe-bb27-47bd-8ddf-f334cac6041a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6805","nonce":117131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":117131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":117131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6807","nonce":117131}]},"public_addr":"172.21.15.135:6805/117131","cluster_addr":"172.21.15.135:6806/117131","heartbeat_back_addr":"172.21.15.135:6808/117131","heartbeat_front_addr":"172.21.15.135:6807/117131","state":["exists","up"]},{"osd":5,"uuid":"7ea449db-9b22-48b8-a61b-1937eeeb6c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6801","nonce":117130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":117130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":117130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6803","nonce":117130}]},"public_addr":"172.21.15.135:6801/117130","cluster_addr":"172.21.15.135:6802/117130","heartbeat_back_addr":"172.21.15.135:6804/117130","heartbeat_front_addr":"172.21.15.135:6803/117130","state":["exists","up"]},{"osd":6,"uuid":"c79b4829-6393-4f9a-a9f8-d2aacb505ae9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6813","nonce":117133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":117133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":117133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6815","nonce":117133}]},"public_addr":"172.21.15.135:6813/117133","cluster_addr":"172.21.15.135:6814/117133","heartbeat_back_addr":"172.21.15.135:6816/117133","heartbeat_front_addr":"172.21.15.135:6815/117133","state":["exists","up"]},{"osd":7,"uuid":"4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6809","nonce":117132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":117132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":117132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6811","nonce":117132}]},"public_addr":"172.21.15.135:6809/117132","cluster_addr":"172.21.15.135:6810/117132","heartbeat_back_addr":"172.21.15.135:6812/117132","heartbeat_front_addr":"172.21.15.135:6811/117132","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:35.977051+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:36.680016+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:35.693147+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:36.368009+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:47:43.418 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.417+0000 7f1bd38da700 10 librados: watch_flush enter 2023-09-30T00:47:43.418 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.417+0000 7f1bd38da700 10 librados: watch_flush exit 2023-09-30T00:47:43.418 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.417+0000 7f1bd38da700 20 client.4239.objecter shutdown clearing up homeless session... 2023-09-30T00:47:43.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.417+0000 7f1bd38da700 10 client.4239.objecter successfully canceled tick 2023-09-30T00:47:43.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.418+0000 7f1bd38da700 1 -- 172.21.15.40:0/3801083530 >> v2:172.21.15.135:6800/116426 conn(0x7f1bb0068800 msgr2=0x7f1bb006acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:43.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.418+0000 7f1bd38da700 1 --2- 172.21.15.40:0/3801083530 >> v2:172.21.15.135:6800/116426 conn(0x7f1bb0068800 0x7f1bb006acb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009270 tx=0x7f1bbc017000 comp rx=0 tx=0).stop 2023-09-30T00:47:43.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.418+0000 7f1bd38da700 10 monclient: shutdown 2023-09-30T00:47:43.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.418+0000 7f1bd38da700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:43.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.418+0000 7f1bd38da700 1 -- 172.21.15.40:0/3801083530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bcc0a5be0 msgr2=0x7f1bcc151420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:43.419 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.418+0000 7f1bd38da700 1 --2- 172.21.15.40:0/3801083530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bcc0a5be0 0x7f1bcc151420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1bc800d8c0 tx=0x7f1bc8008040 comp rx=0 tx=0).stop 2023-09-30T00:47:43.423 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.422+0000 7f1bd38da700 1 -- 172.21.15.40:0/3801083530 shutdown_connections 2023-09-30T00:47:43.423 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.422+0000 7f1bd38da700 1 --2- 172.21.15.40:0/3801083530 >> v2:172.21.15.135:6800/116426 conn(0x7f1bb0068800 0x7f1bb006acb0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:43.423 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.422+0000 7f1bd38da700 1 --2- 172.21.15.40:0/3801083530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bcc13f0e0 0x7f1bcc0aac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:43.423 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.422+0000 7f1bd38da700 1 --2- 172.21.15.40:0/3801083530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1bcc0a5be0 0x7f1bcc151420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:43.423 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.422+0000 7f1bd38da700 1 --2- 172.21.15.40:0/3801083530 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bcc0a5fb0 0x7f1bcc0aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:43.423 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.422+0000 7f1bd38da700 1 -- 172.21.15.40:0/3801083530 >> 172.21.15.40:0/3801083530 conn(0x7f1bcc063280 msgr2=0x7f1bcc0b4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:43.424 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.423+0000 7f1bd38da700 1 -- 172.21.15.40:0/3801083530 shutdown_connections 2023-09-30T00:47:43.424 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.423+0000 7f1bd38da700 1 -- 172.21.15.40:0/3801083530 wait complete. 2023-09-30T00:47:43.424 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:43.423+0000 7f1bd38da700 1 librados: shutdown 2023-09-30T00:47:43.444 DEBUG:teuthology.misc:4 of 8 OSDs are up 2023-09-30T00:47:43.811 INFO:tasks.ceph.osd.3.smithi040.stderr:2023-09-30T00:47:43.807+0000 fe8c700 -1 osd.3 0 waiting for initial osdmap 2023-09-30T00:47:43.819 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-09-30T00:47:43.800+0000 e689700 -1 osd.0 0 waiting for initial osdmap 2023-09-30T00:47:43.822 INFO:tasks.ceph.osd.2.smithi040.stderr:2023-09-30T00:47:43.819+0000 fe8c700 -1 osd.2 0 waiting for initial osdmap 2023-09-30T00:47:44.597 INFO:tasks.ceph.osd.2.smithi040.stderr:2023-09-30T00:47:44.575+0000 14890700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:47:44.611 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-09-30T00:47:44.590+0000 14890700 -1 osd.0 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:47:44.620 INFO:tasks.ceph.osd.3.smithi040.stderr:2023-09-30T00:47:44.605+0000 15091700 -1 osd.3 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:47:47.478 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:01:23.189 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:47:47.561 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:23.247 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:47:47.582 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:01:23.294 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:47:47.589 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:01:23.300 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:47:47.614 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:23.301 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:47:47.639 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:01:23.351 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:47:47.860 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:23.546 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:47:47.914 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:01:23.625 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:47:47.944 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:01:23.656 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:47:50.251 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:47:50.383 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.382+0000 7f2a60fb1700 10 monclient: get_monmap_and_config 2023-09-30T00:47:50.383 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.382+0000 7f2a60fb1700 10 monclient: build_initial_monmap 2023-09-30T00:47:50.383 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.382+0000 7f2a60fb1700 10 monclient: monmap: 2023-09-30T00:47:50.383 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:47:50.384 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:50.384 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:47:50.383819+0000 2023-09-30T00:47:50.384 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:47:50.383819+0000 2023-09-30T00:47:50.384 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:50.384 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:50.384 INFO:teuthology.misc.health.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:50.384 INFO:teuthology.misc.health.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:50.384 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:50.384 INFO:teuthology.misc.health.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:50.384 INFO:teuthology.misc.health.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:50.384 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:50.386 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.383+0000 7f2a60fb1700 1 Processor -- start 2023-09-30T00:47:50.386 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.383+0000 7f2a60fb1700 1 -- start start 2023-09-30T00:47:50.386 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.383+0000 7f2a60fb1700 10 monclient: init 2023-09-30T00:47:50.386 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.384+0000 7f2a60fb1700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:50.386 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.384+0000 7f2a60fb1700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-09-30T00:47:50.386 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.384+0000 7f2a60fb1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a5c0a5be0 0x7f2a5c13eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:50.386 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.384+0000 7f2a60fb1700 10 monclient(hunting): picked mon.noname-b con 0x7f2a5c0a5be0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:50.386 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.384+0000 7f2a60fb1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a5c0a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:50.386 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.384+0000 7f2a60fb1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a5c13f0e0 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:50.386 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.384+0000 7f2a60fb1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a5c0a7750 con 0x7f2a5c13f0e0 2023-09-30T00:47:50.386 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.384+0000 7f2a60fb1700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a5c0a59d0 con 0x7f2a5c0a5fb0 2023-09-30T00:47:50.387 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.384+0000 7f2a60fb1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:50.387 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.384+0000 7f2a60fb1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5c0a56f0 con 0x7f2a5c0a5be0 2023-09-30T00:47:50.387 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.384+0000 7f2a60fb1700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:50.387 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.384+0000 7f2a60fb1700 10 monclient(hunting): authenticate will time out at 5331.685625s 2023-09-30T00:47:50.387 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.385+0000 7f2a59d9b700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f2a5c0a5fb0 0x7f2a5c141d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:45704/0 (socket says 172.21.15.40:45704) 2023-09-30T00:47:50.387 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.385+0000 7f2a59d9b700 1 -- 172.21.15.40:0/3519637870 learned_addr learned my addr 172.21.15.40:0/3519637870 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:50.390 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.389+0000 7f2a5959a700 1 -- 172.21.15.40:0/3519637870 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462312507 0 0) 0x7f2a5c0a7750 con 0x7f2a5c13f0e0 2023-09-30T00:47:50.390 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.389+0000 7f2a5959a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:50.390 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.389+0000 7f2a5959a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:50.390 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.389+0000 7f2a5959a700 10 monclient(hunting): my global_id is 4248 2023-09-30T00:47:50.390 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.389+0000 7f2a5959a700 1 -- 172.21.15.40:0/3519637870 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a44003570 con 0x7f2a5c13f0e0 2023-09-30T00:47:50.391 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.391+0000 7f2a5a59c700 1 --2- 172.21.15.40:0/3519637870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a5c0a5be0 0x7f2a5c13eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:50.392 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.391+0000 7f2a5a59c700 10 monclient(hunting): get_auth_request con 0x7f2a5c0a5be0 auth_method 0 2023-09-30T00:47:50.392 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.391+0000 7f2a5a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:50.392 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.391+0000 7f2a5a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:50.392 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.391+0000 7f2a5a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:50.393 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.392+0000 7f2a5a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:50.393 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.392+0000 7f2a5a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:50.393 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.392+0000 7f2a5a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:50.396 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 -- 172.21.15.40:0/3519637870 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3311570695 0 0) 0x7f2a44003570 con 0x7f2a5c13f0e0 2023-09-30T00:47:50.397 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 -- 172.21.15.40:0/3519637870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a5c0a5be0 msgr2=0x7f2a5c13eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:50.397 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 --2- 172.21.15.40:0/3519637870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a5c0a5be0 0x7f2a5c13eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:50.397 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 -- 172.21.15.40:0/3519637870 >> v1:172.21.15.135:6789/0 conn(0x7f2a5c0a5fb0 legacy=0x7f2a5c141d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:50.397 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient: _finish_hunting 0 2023-09-30T00:47:50.397 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 monclient: found mon.noname-a-legacy 2023-09-30T00:47:50.397 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:50.397 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2023-09-30T00:47:50.397 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 -- 172.21.15.40:0/3519637870 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5c0b1d10 con 0x7f2a5c13f0e0 2023-09-30T00:47:50.397 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient: _finish_auth 0 2023-09-30T00:47:50.397 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:50.397 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 -- 172.21.15.40:0/3519637870 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3033000540 0 0) 0x7f2a540017f0 con 0x7f2a5c13f0e0 2023-09-30T00:47:50.398 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:50.398 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T00:47:50.398 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient: dump: 2023-09-30T00:47:50.398 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:50.398 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:50.398 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:50.398 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:50.398 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:50.398 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:50.398 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:50.398 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:50.399 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:50.399 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:50.399 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2023-09-30T00:47:50.399 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:50.399 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 -- 172.21.15.40:0/3519637870 >> v1:172.21.15.40:6789/0 conn(0x7f2a5c13f0e0 legacy=0x7f2a5c148d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:50.399 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:47:50.399 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 --2- 172.21.15.40:0/3519637870 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a44003a90 0x7f2a44003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:50.399 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient(hunting): picked mon.b con 0x7f2a44003a90 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:50.399 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 --2- 172.21.15.40:0/3519637870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44004810 0x7f2a44004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:50.399 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient(hunting): picked mon.a con 0x7f2a44004810 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:50.399 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 --2- 172.21.15.40:0/3519637870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a5c0a5be0 0x7f2a440051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:50.399 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient(hunting): picked mon.c con 0x7f2a5c0a5be0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:50.400 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:50.400 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 -- 172.21.15.40:0/3519637870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5c0a56f0 con 0x7f2a44004810 2023-09-30T00:47:50.400 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:50.400 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 -- 172.21.15.40:0/3519637870 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4400bb60 con 0x7f2a44003a90 2023-09-30T00:47:50.400 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:50.400 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 1 -- 172.21.15.40:0/3519637870 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4400bd20 con 0x7f2a5c0a5be0 2023-09-30T00:47:50.400 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:50.400 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.395+0000 7f2a5959a700 10 monclient(hunting): _finish_auth 0 2023-09-30T00:47:50.402 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.396+0000 7f2a5a59c700 1 --2- 172.21.15.40:0/3519637870 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a44003a90 0x7f2a44003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:50.402 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.398+0000 7f2a5ad9d700 1 --2- 172.21.15.40:0/3519637870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a5c0a5be0 0x7f2a440051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:50.402 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.398+0000 7f2a5ad9d700 10 monclient(hunting): get_auth_request con 0x7f2a5c0a5be0 auth_method 0 2023-09-30T00:47:50.402 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.398+0000 7f2a5ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:50.402 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.398+0000 7f2a5ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:50.402 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.398+0000 7f2a5ad9d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:47:50.403 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.398+0000 7f2a5a59c700 10 monclient(hunting): get_auth_request con 0x7f2a44003a90 auth_method 0 2023-09-30T00:47:50.403 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.398+0000 7f2a5a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:50.403 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.398+0000 7f2a5a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:50.403 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.398+0000 7f2a5a59c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:47:50.403 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.399+0000 7f2a59d9b700 1 --2- 172.21.15.40:0/3519637870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44004810 0x7f2a44004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:50.403 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.399+0000 7f2a5a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:50.403 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.399+0000 7f2a5a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:50.403 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.399+0000 7f2a5a59c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:47:50.403 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.399+0000 7f2a5ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:50.403 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.399+0000 7f2a5ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:50.403 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.399+0000 7f2a5ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:47:50.403 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.399+0000 7f2a59d9b700 10 monclient(hunting): get_auth_request con 0x7f2a44004810 auth_method 0 2023-09-30T00:47:50.404 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.399+0000 7f2a59d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:50.404 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.399+0000 7f2a59d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:50.404 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.399+0000 7f2a59d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:47:50.404 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.401+0000 7f2a5a59c700 10 monclient(hunting): handle_auth_done global_id 4248 payload 293 2023-09-30T00:47:50.404 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.401+0000 7f2a5a59c700 1 -- 172.21.15.40:0/3519637870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a5c0a5be0 msgr2=0x7f2a440051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:50.404 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.401+0000 7f2a5a59c700 1 --2- 172.21.15.40:0/3519637870 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a5c0a5be0 0x7f2a440051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:50.404 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.401+0000 7f2a5a59c700 1 -- 172.21.15.40:0/3519637870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44004810 msgr2=0x7f2a44004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:50.404 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.401+0000 7f2a5a59c700 1 --2- 172.21.15.40:0/3519637870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44004810 0x7f2a44004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:50.404 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.401+0000 7f2a5a59c700 10 monclient: _finish_hunting 0 2023-09-30T00:47:50.404 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.401+0000 7f2a5a59c700 1 monclient: found mon.b 2023-09-30T00:47:50.404 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.401+0000 7f2a5a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:50.404 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.401+0000 7f2a5a59c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:50.405 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.401+0000 7f2a5a59c700 1 -- 172.21.15.40:0/3519637870 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5c0b1d10 con 0x7f2a44003a90 2023-09-30T00:47:50.405 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.401+0000 7f2a5a59c700 10 monclient: _finish_auth 0 2023-09-30T00:47:50.405 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.401+0000 7f2a5a59c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:50.405 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.401+0000 7f2a60fb1700 5 monclient: authenticate success, global_id 4248 2023-09-30T00:47:50.405 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.401+0000 7f2a60fb1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:50.405 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.403+0000 7f2a5a59c700 1 --2- 172.21.15.40:0/3519637870 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a44003a90 0x7f2a44003ea0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c009c00 tx=0x7f2a4c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=415cbd0a598c0c88 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:50.407 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a5959a700 1 -- 172.21.15.40:0/3519637870 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c00e710 con 0x7f2a44003a90 2023-09-30T00:47:50.408 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a5959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:50.408 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a5959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:50.408 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a5959a700 10 monclient: dump: 2023-09-30T00:47:50.408 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:50.408 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:50.408 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:50.408 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:50.408 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:50.408 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:50.408 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:50.408 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:50.409 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:50.409 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:50.409 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a60fb1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:50.409 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a5959a700 1 -- 172.21.15.40:0/3519637870 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4c00e870 con 0x7f2a44003a90 2023-09-30T00:47:50.409 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a5959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:50.409 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a60fb1700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:50.409 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a60fb1700 4 set_mon_vals no callback set 2023-09-30T00:47:50.409 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a5959a700 1 -- 172.21.15.40:0/3519637870 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c00ec30 con 0x7f2a44003a90 2023-09-30T00:47:50.409 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a5959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:50.409 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a5959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:50.409 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a5959a700 10 monclient: dump: 2023-09-30T00:47:50.409 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:50.410 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:50.410 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:50.410 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:50.410 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:50.410 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:50.410 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:50.410 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:50.410 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:50.410 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:50.412 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a60fb1700 10 monclient: shutdown 2023-09-30T00:47:50.412 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a60fb1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:50.412 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3519637870 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a44003a90 msgr2=0x7f2a44003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:50.412 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a60fb1700 1 --2- 172.21.15.40:0/3519637870 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a44003a90 0x7f2a44003ea0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c009c00 tx=0x7f2a4c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:50.412 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.407+0000 7f2a5ad9d700 1 -- 172.21.15.40:0/3519637870 reap_dead start 2023-09-30T00:47:50.413 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.408+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3519637870 shutdown_connections 2023-09-30T00:47:50.413 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.408+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3519637870 >> 172.21.15.40:0/3519637870 conn(0x7f2a5c063280 msgr2=0x7f2a5c063680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:50.413 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.408+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3519637870 shutdown_connections 2023-09-30T00:47:50.413 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.408+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3519637870 wait complete. 2023-09-30T00:47:50.413 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.408+0000 7f2a60fb1700 10 monclient: build_initial_monmap 2023-09-30T00:47:50.413 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.408+0000 7f2a60fb1700 10 monclient: monmap: 2023-09-30T00:47:50.413 INFO:teuthology.misc.health.smithi040.stderr:epoch 0 2023-09-30T00:47:50.413 INFO:teuthology.misc.health.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:50.413 INFO:teuthology.misc.health.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:50.413 INFO:teuthology.misc.health.smithi040.stderr:created 0.000000 2023-09-30T00:47:50.413 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:50.413 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:50.414 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:50.414 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:50.414 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:50.414 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:50.414 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.409+0000 7f2a60fb1700 1 librados: starting msgr at 2023-09-30T00:47:50.414 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.409+0000 7f2a60fb1700 1 librados: starting objecter 2023-09-30T00:47:50.414 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.409+0000 7f2a60fb1700 1 Processor -- start 2023-09-30T00:47:50.414 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.409+0000 7f2a60fb1700 1 -- start start 2023-09-30T00:47:50.414 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.409+0000 7f2a60fb1700 1 librados: setting wanted keys 2023-09-30T00:47:50.414 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.409+0000 7f2a60fb1700 1 librados: calling monclient init 2023-09-30T00:47:50.415 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.409+0000 7f2a60fb1700 10 monclient: init 2023-09-30T00:47:50.415 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:50.415 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:47:50.415 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44003a90 0x7f2a5c0aa440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:50.415 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 10 monclient(hunting): picked mon.noname-a con 0x7f2a44003a90 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:50.415 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a5c0b50d0 0x7f2a5c138d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:50.415 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 10 monclient(hunting): picked mon.noname-b con 0x7f2a5c0b50d0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:50.415 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a5c139250 0x7f2a5c14e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:50.415 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 10 monclient(hunting): picked mon.noname-c con 0x7f2a5c139250 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:50.415 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:50.415 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5c0abd40 con 0x7f2a44003a90 2023-09-30T00:47:50.415 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:50.416 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5c0aaaf0 con 0x7f2a5c0b50d0 2023-09-30T00:47:50.416 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:50.416 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5c1396c0 con 0x7f2a5c139250 2023-09-30T00:47:50.416 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:50.416 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.410+0000 7f2a60fb1700 10 monclient(hunting): authenticate will time out at 5331.711423s 2023-09-30T00:47:50.416 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.414+0000 7f2a5ad9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a5c139250 0x7f2a5c14e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:50.462 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.461+0000 7f2a59d9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a5c0b50d0 0x7f2a5c138d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:50.462 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.461+0000 7f2a59d9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a5c0b50d0 0x7f2a5c138d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:33850/0 (socket says 172.21.15.40:33850) 2023-09-30T00:47:50.462 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.461+0000 7f2a59d9b700 1 -- 172.21.15.40:0/3086171603 learned_addr learned my addr 172.21.15.40:0/3086171603 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:50.462 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.461+0000 7f2a59d9b700 10 monclient(hunting): get_auth_request con 0x7f2a5c0b50d0 auth_method 0 2023-09-30T00:47:50.462 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.461+0000 7f2a59d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:50.462 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.461+0000 7f2a59d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:50.462 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.461+0000 7f2a59d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:50.463 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.462+0000 7f2a59d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:50.463 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.462+0000 7f2a59d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:50.463 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.462+0000 7f2a59d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:50.466 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.465+0000 7f2a59d9b700 10 monclient(hunting): handle_auth_done global_id 4276 payload 824 2023-09-30T00:47:50.466 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.465+0000 7f2a59d9b700 1 -- 172.21.15.40:0/3086171603 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a5c139250 msgr2=0x7f2a5c14e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:50.466 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.465+0000 7f2a59d9b700 1 --2- 172.21.15.40:0/3086171603 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a5c139250 0x7f2a5c14e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:50.466 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.465+0000 7f2a59d9b700 1 -- 172.21.15.40:0/3086171603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44003a90 msgr2=0x7f2a5c0aa440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:50.466 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.465+0000 7f2a59d9b700 1 --2- 172.21.15.40:0/3086171603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44003a90 0x7f2a5c0aa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:50.466 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.465+0000 7f2a59d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:47:50.466 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.465+0000 7f2a59d9b700 1 monclient: found mon.noname-b 2023-09-30T00:47:50.466 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.465+0000 7f2a59d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:50.466 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.465+0000 7f2a59d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:47:50.467 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.465+0000 7f2a59d9b700 1 -- 172.21.15.40:0/3086171603 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5c0abf40 con 0x7f2a5c0b50d0 2023-09-30T00:47:50.468 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.467+0000 7f2a59d9b700 1 --2- 172.21.15.40:0/3086171603 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a5c0b50d0 0x7f2a5c138d10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2a50001530 tx=0x7f2a500058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:50.468 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.467+0000 7f2a4a7fc700 10 client.?.objecter ms_handle_connect 0x7f2a5c0b50d0 2023-09-30T00:47:50.468 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.467+0000 7f2a4a7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:50.473 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a4a7fc700 1 -- 172.21.15.40:0/3086171603 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a50010520 con 0x7f2a5c0b50d0 2023-09-30T00:47:50.473 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a4a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:50.473 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a4a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:50.473 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a4a7fc700 10 monclient: dump: 2023-09-30T00:47:50.473 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:50.473 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:50.473 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:50.473 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:50.473 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:50.474 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:50.474 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:50.474 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:50.474 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:50.474 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:50.474 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a4a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:47:50.474 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a4a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:50.474 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a4a7fc700 1 -- 172.21.15.40:0/3086171603 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a50010680 con 0x7f2a5c0b50d0 2023-09-30T00:47:50.474 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a60fb1700 5 monclient: authenticate success, global_id 4276 2023-09-30T00:47:50.474 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a60fb1700 10 monclient: _renew_subs 2023-09-30T00:47:50.474 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a60fb1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:50.474 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3086171603 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a5c0a8c30 con 0x7f2a5c0b50d0 2023-09-30T00:47:50.475 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a4a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:50.475 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a4a7fc700 1 -- 172.21.15.40:0/3086171603 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a50010a40 con 0x7f2a5c0b50d0 2023-09-30T00:47:50.475 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a4a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:50.475 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a4a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:50.475 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a4a7fc700 10 monclient: dump: 2023-09-30T00:47:50.475 INFO:teuthology.misc.health.smithi040.stderr:epoch 1 2023-09-30T00:47:50.475 INFO:teuthology.misc.health.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:50.475 INFO:teuthology.misc.health.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:50.475 INFO:teuthology.misc.health.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:50.475 INFO:teuthology.misc.health.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:50.475 INFO:teuthology.misc.health.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:50.476 INFO:teuthology.misc.health.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:50.476 INFO:teuthology.misc.health.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:50.476 INFO:teuthology.misc.health.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:50.476 INFO:teuthology.misc.health.smithi040.stderr: 2023-09-30T00:47:50.476 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.472+0000 7f2a60fb1700 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:50.476 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.472+0000 7f2a60fb1700 10 monclient: _renew_subs 2023-09-30T00:47:50.476 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.472+0000 7f2a60fb1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:50.476 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.472+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3086171603 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a5c14eb00 con 0x7f2a5c0b50d0 2023-09-30T00:47:50.476 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.472+0000 7f2a60fb1700 1 librados: init done 2023-09-30T00:47:50.476 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.471+0000 7f2a58d99700 4 set_mon_vals no callback set 2023-09-30T00:47:50.477 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.473+0000 7f2a60fb1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:50.477 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.473+0000 7f2a60fb1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:50.477 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.473+0000 7f2a60fb1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:50.477 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.473+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3086171603 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a5c064a00 con 0x7f2a5c0b50d0 2023-09-30T00:47:50.728 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.727+0000 7f2a4a7fc700 1 -- 172.21.15.40:0/3086171603 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f2a500133e0 con 0x7f2a5c0b50d0 2023-09-30T00:47:50.729 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.727+0000 7f2a4a7fc700 1 --2- 172.21.15.40:0/3086171603 >> v2:172.21.15.135:6800/116426 conn(0x7f2a3c0688d0 0x7f2a3c06ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:50.729 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.727+0000 7f2a4a7fc700 1 -- 172.21.15.40:0/3086171603 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3733+0+0 (secure 0 0 0) 0x7f2a50083c60 con 0x7f2a5c0b50d0 2023-09-30T00:47:50.729 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.727+0000 7f2a4a7fc700 10 client.4276.objecter ms_dispatch 0x7f2a5c001550 osd_map(21..21 src has 1..21) v4 2023-09-30T00:47:50.729 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.727+0000 7f2a4a7fc700 3 client.4276.objecter handle_osd_map got epochs [21,21] > 0 2023-09-30T00:47:50.729 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.727+0000 7f2a4a7fc700 3 client.4276.objecter handle_osd_map decoding full epoch 21 2023-09-30T00:47:50.729 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.728+0000 7f2a4a7fc700 20 client.4276.objecter dump_active .. 0 homeless 2023-09-30T00:47:50.729 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.728+0000 7f2a5a59c700 1 --2- 172.21.15.40:0/3086171603 >> v2:172.21.15.135:6800/116426 conn(0x7f2a3c0688d0 0x7f2a3c06ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:50.729 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.728+0000 7f2a5a59c700 10 monclient: get_auth_request con 0x7f2a3c0688d0 auth_method 0 2023-09-30T00:47:50.730 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.728+0000 7f2a4a7fc700 10 client.4276.objecter ms_handle_connect 0x7f2a3c0688d0 2023-09-30T00:47:50.730 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.728+0000 7f2a5a59c700 1 --2- 172.21.15.40:0/3086171603 >> v2:172.21.15.135:6800/116426 conn(0x7f2a3c0688d0 0x7f2a3c06ad80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c0091e0 tx=0x7f2a4c01a000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:50.743 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.742+0000 7f2a4a7fc700 1 -- 172.21.15.40:0/3086171603 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2a500517c0 con 0x7f2a5c0b50d0 2023-09-30T00:47:50.743 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.742+0000 7f2a4a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:50.743 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:50.742+0000 7f2a4a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:51.026 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.026+0000 7f2a60fb1700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:51.027 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.026+0000 7f2a60fb1700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:51.027 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.026+0000 7f2a60fb1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:51.027 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.026+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3086171603 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a5c0a63e0 con 0x7f2a5c0b50d0 2023-09-30T00:47:51.067 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.066+0000 7f2a4a7fc700 1 -- 172.21.15.40:0/3086171603 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+10417 (secure 0 0 0) 0x7f2a50019090 con 0x7f2a5c0b50d0 2023-09-30T00:47:51.067 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.066+0000 7f2a4a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:51.067 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.066+0000 7f2a4a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:51.067 INFO:teuthology.misc.health.smithi040.stdout: 2023-09-30T00:47:51.068 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":21,"fsid":"238dba7c-9190-48bd-bb67-322ebc9ecbd0","created":"2023-09-30T00:46:45.822318+0000","modified":"2023-09-30T00:47:46.989307+0000","last_up_change":"2023-09-30T00:47:45.793188+0000","last_in_change":"2023-09-30T00:46:52.657791+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:47:40.523068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"66e1da65-310e-4463-8aa0-431896832e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":117238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":117238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":117238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":117238}]},"public_addr":"172.21.15.40:6804/117238","cluster_addr":"172.21.15.40:6805/117238","heartbeat_back_addr":"172.21.15.40:6807/117238","heartbeat_front_addr":"172.21.15.40:6806/117238","state":["exists","up"]},{"osd":1,"uuid":"d6dd269f-2eda-4ef7-9da1-2ed5b16291b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":117237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6801","nonce":117237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":117237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":117237}]},"public_addr":"172.21.15.40:6800/117237","cluster_addr":"172.21.15.40:6801/117237","heartbeat_back_addr":"172.21.15.40:6803/117237","heartbeat_front_addr":"172.21.15.40:6802/117237","state":["exists","up"]},{"osd":2,"uuid":"8f0de27d-3674-4ff4-8451-4fcc90786c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":117239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":117239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":117239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":117239}]},"public_addr":"172.21.15.40:6808/117239","cluster_addr":"172.21.15.40:6809/117239","heartbeat_back_addr":"172.21.15.40:6811/117239","heartbeat_front_addr":"172.21.15.40:6810/117239","state":["exists","up"]},{"osd":3,"uuid":"d18ec2f5-7e8f-4732-a523-adc495115cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":117236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":117236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":117236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":117236}]},"public_addr":"172.21.15.40:6812/117236","cluster_addr":"172.21.15.40:6813/117236","heartbeat_back_addr":"172.21.15.40:6815/117236","heartbeat_front_addr":"172.21.15.40:6814/117236","state":["exists","up"]},{"osd":4,"uuid":"4e2465fe-bb27-47bd-8ddf-f334cac6041a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6805","nonce":117131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":117131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":117131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6807","nonce":117131}]},"public_addr":"172.21.15.135:6805/117131","cluster_addr":"172.21.15.135:6806/117131","heartbeat_back_addr":"172.21.15.135:6808/117131","heartbeat_front_addr":"172.21.15.135:6807/117131","state":["exists","up"]},{"osd":5,"uuid":"7ea449db-9b22-48b8-a61b-1937eeeb6c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6801","nonce":117130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":117130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":117130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6803","nonce":117130}]},"public_addr":"172.21.15.135:6801/117130","cluster_addr":"172.21.15.135:6802/117130","heartbeat_back_addr":"172.21.15.135:6804/117130","heartbeat_front_addr":"172.21.15.135:6803/117130","state":["exists","up"]},{"osd":6,"uuid":"c79b4829-6393-4f9a-a9f8-d2aacb505ae9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6813","nonce":117133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":117133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":117133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6815","nonce":117133}]},"public_addr":"172.21.15.135:6813/117133","cluster_addr":"172.21.15.135:6814/117133","heartbeat_back_addr":"172.21.15.135:6816/117133","heartbeat_front_addr":"172.21.15.135:6815/117133","state":["exists","up"]},{"osd":7,"uuid":"4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6809","nonce":117132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":117132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":117132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6811","nonce":117132}]},"public_addr":"172.21.15.135:6809/117132","cluster_addr":"172.21.15.135:6810/117132","heartbeat_back_addr":"172.21.15.135:6812/117132","heartbeat_front_addr":"172.21.15.135:6811/117132","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.725062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.291589+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.713534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.731764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:35.977051+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:36.680016+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:35.693147+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:36.368009+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:47:51.070 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.068+0000 7f2a60fb1700 10 librados: watch_flush enter 2023-09-30T00:47:51.070 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 10 librados: watch_flush exit 2023-09-30T00:47:51.070 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 20 client.4276.objecter shutdown clearing up homeless session... 2023-09-30T00:47:51.070 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 10 client.4276.objecter successfully canceled tick 2023-09-30T00:47:51.070 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3086171603 >> v2:172.21.15.135:6800/116426 conn(0x7f2a3c0688d0 msgr2=0x7f2a3c06ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:51.070 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 1 --2- 172.21.15.40:0/3086171603 >> v2:172.21.15.135:6800/116426 conn(0x7f2a3c0688d0 0x7f2a3c06ad80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c0091e0 tx=0x7f2a4c01a000 comp rx=0 tx=0).stop 2023-09-30T00:47:51.070 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 10 monclient: shutdown 2023-09-30T00:47:51.070 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:51.070 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3086171603 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a5c0b50d0 msgr2=0x7f2a5c138d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:51.071 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 1 --2- 172.21.15.40:0/3086171603 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a5c0b50d0 0x7f2a5c138d10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2a50001530 tx=0x7f2a500058d0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.071 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3086171603 shutdown_connections 2023-09-30T00:47:51.071 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 1 --2- 172.21.15.40:0/3086171603 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2a5c139250 0x7f2a5c14e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.071 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 1 --2- 172.21.15.40:0/3086171603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a44003a90 0x7f2a5c0aa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.071 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 1 --2- 172.21.15.40:0/3086171603 >> v2:172.21.15.135:6800/116426 conn(0x7f2a3c0688d0 0x7f2a3c06ad80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.071 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 1 --2- 172.21.15.40:0/3086171603 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2a5c0b50d0 0x7f2a5c138d10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.071 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3086171603 >> 172.21.15.40:0/3086171603 conn(0x7f2a5c063280 msgr2=0x7f2a5c13ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:51.071 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3086171603 shutdown_connections 2023-09-30T00:47:51.071 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 1 -- 172.21.15.40:0/3086171603 wait complete. 2023-09-30T00:47:51.071 INFO:teuthology.misc.health.smithi040.stderr:2023-09-30T00:47:51.069+0000 7f2a60fb1700 1 librados: shutdown 2023-09-30T00:47:51.084 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-09-30T00:47:51.084 INFO:tasks.ceph:Creating RBD pool 2023-09-30T00:47:51.084 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-30T00:47:51.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.217+0000 7f84de446700 10 monclient: get_monmap_and_config 2023-09-30T00:47:51.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.217+0000 7f84de446700 10 monclient: build_initial_monmap 2023-09-30T00:47:51.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.217+0000 7f84de446700 10 monclient: monmap: 2023-09-30T00:47:51.218 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:51.218 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:51.219 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:51.218670+0000 2023-09-30T00:47:51.219 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:51.218670+0000 2023-09-30T00:47:51.219 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:51.219 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:51.219 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:51.219 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:51.219 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:51.219 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:51.219 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:51.219 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.218+0000 7f84de446700 1 Processor -- start 2023-09-30T00:47:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.218+0000 7f84de446700 1 -- start start 2023-09-30T00:47:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.218+0000 7f84de446700 10 monclient: init 2023-09-30T00:47:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84de446700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84de446700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-09-30T00:47:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84de446700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84d80b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84de446700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f84d80b1830 0x7f84d80a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84de446700 10 monclient(hunting): picked mon.noname-a con 0x7f84d80b1830 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84de446700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84d813ac20 0x7f84d813b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84de446700 10 monclient(hunting): picked mon.noname-b con 0x7f84d813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84de446700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d80a5b20 con 0x7f84d80b1460 2023-09-30T00:47:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84de446700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84de446700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d8141e40 con 0x7f84d80b1830 2023-09-30T00:47:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84de446700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84de446700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d80a64f0 con 0x7f84d813ac20 2023-09-30T00:47:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84de446700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84de446700 10 monclient(hunting): authenticate will time out at 5332.520278s 2023-09-30T00:47:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.219+0000 7f84dc9e3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84d813ac20 0x7f84d813b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.220+0000 7f84dc9e3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84d813ac20 0x7f84d813b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:57002/0 (socket says 172.21.15.40:57002) 2023-09-30T00:47:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.220+0000 7f84dc9e3700 1 -- 172.21.15.40:0/2140854180 learned_addr learned my addr 172.21.15.40:0/2140854180 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:51.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.220+0000 7f84dc9e3700 10 monclient(hunting): get_auth_request con 0x7f84d813ac20 auth_method 0 2023-09-30T00:47:51.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.220+0000 7f84dc9e3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:51.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.220+0000 7f84dc9e3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:51.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.220+0000 7f84dc9e3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:51.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.221+0000 7f84dc9e3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:51.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.221+0000 7f84dc9e3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:51.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.221+0000 7f84dc9e3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:51.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.223+0000 7f84dc9e3700 10 monclient(hunting): handle_auth_done global_id 4217 payload 274 2023-09-30T00:47:51.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.223+0000 7f84dc9e3700 1 -- 172.21.15.40:0/2140854180 >> v2:172.21.15.40:3300/0 conn(0x7f84d80b1830 msgr2=0x7f84d80a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:51.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.223+0000 7f84dc9e3700 1 --2- 172.21.15.40:0/2140854180 >> v2:172.21.15.40:3300/0 conn(0x7f84d80b1830 0x7f84d80a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.223+0000 7f84dc9e3700 1 -- 172.21.15.40:0/2140854180 >> v1:172.21.15.40:6789/0 conn(0x7f84d80b1460 legacy=0x7f84d8141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:51.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.223+0000 7f84dc9e3700 10 monclient: _finish_hunting 0 2023-09-30T00:47:51.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.223+0000 7f84dc9e3700 1 monclient: found mon.noname-b 2023-09-30T00:47:51.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.223+0000 7f84dc9e3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:51.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.223+0000 7f84dc9e3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:47:51.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.223+0000 7f84dc9e3700 1 -- 172.21.15.40:0/2140854180 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d80a71f0 con 0x7f84d813ac20 2023-09-30T00:47:51.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.226+0000 7f84dc9e3700 1 --2- 172.21.15.40:0/2140854180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84d813ac20 0x7f84d813b080 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009f20 tx=0x7f84d0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbd1a31591113acd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:51.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.232+0000 7f84d6ffd700 1 -- 172.21.15.40:0/2140854180 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d000e070 con 0x7f84d813ac20 2023-09-30T00:47:51.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.232+0000 7f84d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:51.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84d6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:51.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84d6ffd700 10 monclient: dump: 2023-09-30T00:47:51.234 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:51.234 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:51.234 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:51.234 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:51.234 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:51.234 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:51.234 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:51.234 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:51.234 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:51.234 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:51.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84d6ffd700 10 monclient: _finish_auth 0 2023-09-30T00:47:51.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84d6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:51.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84d6ffd700 1 -- 172.21.15.40:0/2140854180 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d000ba40 con 0x7f84d813ac20 2023-09-30T00:47:51.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84de446700 5 monclient: authenticate success, global_id 4217 2023-09-30T00:47:51.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84d6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:51.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84de446700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:51.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84d6ffd700 1 -- 172.21.15.40:0/2140854180 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d00092e0 con 0x7f84d813ac20 2023-09-30T00:47:51.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84d6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:51.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84de446700 4 set_mon_vals no callback set 2023-09-30T00:47:51.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84d6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:51.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84d6ffd700 10 monclient: dump: 2023-09-30T00:47:51.236 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:51.236 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:51.236 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:51.236 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:51.236 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:51.236 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:51.236 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:51.236 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:51.236 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:51.236 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:51.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84de446700 10 monclient: shutdown 2023-09-30T00:47:51.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84de446700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:51.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84de446700 1 -- 172.21.15.40:0/2140854180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84d813ac20 msgr2=0x7f84d813b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:51.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84de446700 1 --2- 172.21.15.40:0/2140854180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84d813ac20 0x7f84d813b080 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009f20 tx=0x7f84d0009840 comp rx=0 tx=0).stop 2023-09-30T00:47:51.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84de446700 1 -- 172.21.15.40:0/2140854180 shutdown_connections 2023-09-30T00:47:51.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84de446700 1 --2- 172.21.15.40:0/2140854180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84d813ac20 0x7f84d813b080 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.240 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84de446700 1 --2- 172.21.15.40:0/2140854180 >> v2:172.21.15.40:3300/0 conn(0x7f84d80b1830 0x7f84d80a5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.240 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84de446700 1 -- 172.21.15.40:0/2140854180 >> 172.21.15.40:0/2140854180 conn(0x7f84d80631f0 msgr2=0x7f84d80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:51.240 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.233+0000 7f84de446700 1 -- 172.21.15.40:0/2140854180 shutdown_connections 2023-09-30T00:47:51.240 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.234+0000 7f84de446700 1 -- 172.21.15.40:0/2140854180 wait complete. 2023-09-30T00:47:51.240 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.234+0000 7f84de446700 10 monclient: build_initial_monmap 2023-09-30T00:47:51.240 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.234+0000 7f84de446700 10 monclient: monmap: 2023-09-30T00:47:51.240 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:51.240 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:51.240 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:51.240 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:51.240 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:51.241 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:51.241 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:51.241 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:51.241 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:51.241 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:51.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.234+0000 7f84de446700 1 librados: starting msgr at 2023-09-30T00:47:51.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.234+0000 7f84de446700 1 librados: starting objecter 2023-09-30T00:47:51.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.235+0000 7f84de446700 1 Processor -- start 2023-09-30T00:47:51.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.235+0000 7f84de446700 1 -- start start 2023-09-30T00:47:51.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.235+0000 7f84de446700 1 librados: setting wanted keys 2023-09-30T00:47:51.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.235+0000 7f84de446700 1 librados: calling monclient init 2023-09-30T00:47:51.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.235+0000 7f84de446700 10 monclient: init 2023-09-30T00:47:51.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.235+0000 7f84de446700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:51.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.235+0000 7f84de446700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:47:51.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.235+0000 7f84de446700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84d80b1460 0x7f84d80a8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:51.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.235+0000 7f84de446700 10 monclient(hunting): picked mon.noname-a con 0x7f84d80b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:51.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.235+0000 7f84de446700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84d80b1830 0x7f84d81398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:51.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.235+0000 7f84de446700 10 monclient(hunting): picked mon.noname-c con 0x7f84d80b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:51.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84de446700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84d813ac20 0x7f84d80a7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:51.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84de446700 10 monclient(hunting): picked mon.noname-b con 0x7f84d813ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:51.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84de446700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:51.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84de446700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d80a97c0 con 0x7f84d80b1460 2023-09-30T00:47:51.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84de446700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:51.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84de446700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d80ae7c0 con 0x7f84d813ac20 2023-09-30T00:47:51.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84de446700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:51.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84de446700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d80a9450 con 0x7f84d80b1830 2023-09-30T00:47:51.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84de446700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:51.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84de446700 10 monclient(hunting): authenticate will time out at 5332.537048s 2023-09-30T00:47:51.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84d7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84d80b1460 0x7f84d80a8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:51.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84d7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84d80b1460 0x7f84d80a8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44432/0 (socket says 172.21.15.40:44432) 2023-09-30T00:47:51.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84d7fff700 1 -- 172.21.15.40:0/2447755799 learned_addr learned my addr 172.21.15.40:0/2447755799 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:51.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84d7fff700 10 monclient(hunting): get_auth_request con 0x7f84d80b1460 auth_method 0 2023-09-30T00:47:51.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84d7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:51.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84d7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:51.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.236+0000 7f84d7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:51.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.237+0000 7f84dc9e3700 1 --2- 172.21.15.40:0/2447755799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84d813ac20 0x7f84d80a7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:51.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.237+0000 7f84dc9e3700 10 monclient(hunting): get_auth_request con 0x7f84d813ac20 auth_method 0 2023-09-30T00:47:51.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.237+0000 7f84dc9e3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:51.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.237+0000 7f84dc9e3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:51.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.237+0000 7f84dc9e3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:51.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.237+0000 7f84d7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:51.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.237+0000 7f84d7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:51.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.237+0000 7f84d7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:51.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.238+0000 7f84dc9e3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:51.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.238+0000 7f84dc9e3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:51.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.238+0000 7f84dc9e3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:51.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.238+0000 7f84d77fe700 1 --2- 172.21.15.40:0/2447755799 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84d80b1830 0x7f84d81398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:51.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.239+0000 7f84d77fe700 10 monclient(hunting): get_auth_request con 0x7f84d80b1830 auth_method 0 2023-09-30T00:47:51.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.239+0000 7f84d77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:51.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.239+0000 7f84d77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:51.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.239+0000 7f84d77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:51.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.239+0000 7f84d77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:51.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.239+0000 7f84d77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:51.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.239+0000 7f84d77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:51.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.241+0000 7f84d7fff700 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2023-09-30T00:47:51.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.241+0000 7f84d7fff700 1 -- 172.21.15.40:0/2447755799 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84d80b1830 msgr2=0x7f84d81398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:51.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.241+0000 7f84d7fff700 1 --2- 172.21.15.40:0/2447755799 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84d80b1830 0x7f84d81398c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.241+0000 7f84d7fff700 1 -- 172.21.15.40:0/2447755799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84d813ac20 msgr2=0x7f84d80a7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:51.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.241+0000 7f84d7fff700 1 --2- 172.21.15.40:0/2447755799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84d813ac20 0x7f84d80a7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.241+0000 7f84d7fff700 10 monclient: _finish_hunting 0 2023-09-30T00:47:51.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.241+0000 7f84d7fff700 1 monclient: found mon.noname-a 2023-09-30T00:47:51.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.241+0000 7f84d7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:51.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.241+0000 7f84d7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:51.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.241+0000 7f84d7fff700 1 -- 172.21.15.40:0/2447755799 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d0009ae0 con 0x7f84d80b1460 2023-09-30T00:47:51.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.241+0000 7f84dc9e3700 1 --2- 172.21.15.40:0/2447755799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84d813ac20 0x7f84d80a7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:51.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.245+0000 7f84d7fff700 1 --2- 172.21.15.40:0/2447755799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84d80b1460 0x7f84d80a8f10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f84c800b600 tx=0x7f84c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:51.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.245+0000 7f84bffff700 10 client.?.objecter ms_handle_connect 0x7f84d80b1460 2023-09-30T00:47:51.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.245+0000 7f84bffff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:51.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.247+0000 7f84bffff700 1 -- 172.21.15.40:0/2447755799 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c800f510 con 0x7f84d80b1460 2023-09-30T00:47:51.249 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.247+0000 7f84bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:51.249 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.247+0000 7f84bffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:51.249 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.247+0000 7f84bffff700 10 monclient: dump: 2023-09-30T00:47:51.249 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:51.249 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:51.249 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:51.249 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:51.249 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:51.249 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:51.249 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:51.250 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:51.250 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:51.250 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:51.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.247+0000 7f84bffff700 10 monclient: _finish_auth 0 2023-09-30T00:47:51.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.247+0000 7f84bffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:51.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.247+0000 7f84de446700 5 monclient: authenticate success, global_id 4251 2023-09-30T00:47:51.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.247+0000 7f84de446700 10 monclient: _renew_subs 2023-09-30T00:47:51.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.247+0000 7f84de446700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:51.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.247+0000 7f84de446700 1 -- 172.21.15.40:0/2447755799 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d813ef70 con 0x7f84d80b1460 2023-09-30T00:47:51.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.248+0000 7f84de446700 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:51.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.248+0000 7f84de446700 10 monclient: _renew_subs 2023-09-30T00:47:51.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.248+0000 7f84de446700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:51.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.248+0000 7f84de446700 1 -- 172.21.15.40:0/2447755799 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d80aa310 con 0x7f84d80b1460 2023-09-30T00:47:51.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.248+0000 7f84de446700 1 librados: init done 2023-09-30T00:47:51.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.249+0000 7f84de446700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:51.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.249+0000 7f84de446700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:51.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.249+0000 7f84de446700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:51.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.249+0000 7f84de446700 1 -- 172.21.15.40:0/2447755799 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d80b07e0 con 0x7f84d80b1460 2023-09-30T00:47:51.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.250+0000 7f84bffff700 1 -- 172.21.15.40:0/2447755799 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c800f880 con 0x7f84d80b1460 2023-09-30T00:47:51.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.250+0000 7f84bffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:51.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.250+0000 7f84bffff700 1 -- 172.21.15.40:0/2447755799 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c800fa10 con 0x7f84d80b1460 2023-09-30T00:47:51.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.250+0000 7f84bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:51.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.250+0000 7f84bffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:51.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.250+0000 7f84bffff700 10 monclient: dump: 2023-09-30T00:47:51.252 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:51.252 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:51.253 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:51.253 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:51.253 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:51.253 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:51.253 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:51.253 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:51.253 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:51.253 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:51.253 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.250+0000 7f84d67fc700 4 set_mon_vals no callback set 2023-09-30T00:47:51.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.553+0000 7f84bffff700 1 -- 172.21.15.40:0/2447755799 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f84c800fb70 con 0x7f84d80b1460 2023-09-30T00:47:51.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.553+0000 7f84bffff700 1 --2- 172.21.15.40:0/2447755799 >> v2:172.21.15.135:6800/116426 conn(0x7f84c4068800 0x7f84c406acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.553+0000 7f84bffff700 1 -- 172.21.15.40:0/2447755799 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3733+0+0 (secure 0 0 0) 0x7f84c8084030 con 0x7f84d80b1460 2023-09-30T00:47:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.553+0000 7f84bffff700 10 client.4251.objecter ms_dispatch 0x7f84d8001550 osd_map(21..21 src has 1..21) v4 2023-09-30T00:47:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.553+0000 7f84bffff700 3 client.4251.objecter handle_osd_map got epochs [21,21] > 0 2023-09-30T00:47:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.553+0000 7f84bffff700 3 client.4251.objecter handle_osd_map decoding full epoch 21 2023-09-30T00:47:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.553+0000 7f84bffff700 20 client.4251.objecter dump_active .. 0 homeless 2023-09-30T00:47:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.554+0000 7f84bffff700 1 -- 172.21.15.40:0/2447755799 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f84c80843d0 con 0x7f84d80b1460 2023-09-30T00:47:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.554+0000 7f84bffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:51.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.554+0000 7f84bffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:51.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.554+0000 7f84d77fe700 1 --2- 172.21.15.40:0/2447755799 >> v2:172.21.15.135:6800/116426 conn(0x7f84c4068800 0x7f84c406acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:51.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.554+0000 7f84d77fe700 10 monclient: get_auth_request con 0x7f84c4068800 auth_method 0 2023-09-30T00:47:51.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.554+0000 7f84bffff700 10 client.4251.objecter ms_handle_connect 0x7f84c4068800 2023-09-30T00:47:51.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.554+0000 7f84d77fe700 1 --2- 172.21.15.40:0/2447755799 >> v2:172.21.15.135:6800/116426 conn(0x7f84c4068800 0x7f84c406acb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f84d80aaee0 tx=0x7f84cc0091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:51.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.774+0000 7f84de446700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T00:47:51.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.774+0000 7f84de446700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T00:47:51.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.774+0000 7f84de446700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:51.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:51.774+0000 7f84de446700 1 -- 172.21.15.40:0/2447755799 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f84d8064a00 con 0x7f84d80b1460 2023-09-30T00:47:52.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.081+0000 7f84bffff700 1 -- 172.21.15.40:0/2447755799 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f84c80515a0 con 0x7f84d80b1460 2023-09-30T00:47:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.390+0000 7f84bffff700 1 -- 172.21.15.40:0/2447755799 <== mon.0 v2:172.21.15.40:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v22) v1 ==== 109+0+0 (secure 0 0 0) 0x7f84c8055110 con 0x7f84d80b1460 2023-09-30T00:47:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.390+0000 7f84bffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T00:47:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.390+0000 7f84bffff700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-09-30T00:47:52.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.394+0000 7f84de446700 10 librados: watch_flush enter 2023-09-30T00:47:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.394+0000 7f84de446700 10 librados: watch_flush exit 2023-09-30T00:47:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.394+0000 7f84de446700 20 client.4251.objecter shutdown clearing up homeless session... 2023-09-30T00:47:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.394+0000 7f84de446700 10 client.4251.objecter successfully canceled tick 2023-09-30T00:47:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.395+0000 7f84de446700 1 -- 172.21.15.40:0/2447755799 >> v2:172.21.15.135:6800/116426 conn(0x7f84c4068800 msgr2=0x7f84c406acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.395+0000 7f84de446700 1 --2- 172.21.15.40:0/2447755799 >> v2:172.21.15.135:6800/116426 conn(0x7f84c4068800 0x7f84c406acb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f84d80aaee0 tx=0x7f84cc0091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.395+0000 7f84de446700 10 monclient: shutdown 2023-09-30T00:47:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.395+0000 7f84de446700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.395+0000 7f84de446700 1 -- 172.21.15.40:0/2447755799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84d80b1460 msgr2=0x7f84d80a8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:52.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.395+0000 7f84de446700 1 --2- 172.21.15.40:0/2447755799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84d80b1460 0x7f84d80a8f10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f84c800b600 tx=0x7f84c80091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.400+0000 7f84de446700 1 -- 172.21.15.40:0/2447755799 shutdown_connections 2023-09-30T00:47:52.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.400+0000 7f84de446700 1 --2- 172.21.15.40:0/2447755799 >> v2:172.21.15.135:6800/116426 conn(0x7f84c4068800 0x7f84c406acb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.400+0000 7f84de446700 1 --2- 172.21.15.40:0/2447755799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f84d813ac20 0x7f84d80a7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.400+0000 7f84de446700 1 --2- 172.21.15.40:0/2447755799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84d80b1460 0x7f84d80a8f10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.400+0000 7f84de446700 1 --2- 172.21.15.40:0/2447755799 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f84d80b1830 0x7f84d81398c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.400+0000 7f84de446700 1 -- 172.21.15.40:0/2447755799 >> 172.21.15.40:0/2447755799 conn(0x7f84d80631f0 msgr2=0x7f84d8141130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.400+0000 7f84de446700 1 -- 172.21.15.40:0/2447755799 shutdown_connections 2023-09-30T00:47:52.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.400+0000 7f84de446700 1 -- 172.21.15.40:0/2447755799 wait complete. 2023-09-30T00:47:52.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.404+0000 7f84de446700 1 librados: shutdown 2023-09-30T00:47:52.406 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'rbd' created 2023-09-30T00:47:52.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-30T00:47:52.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.579+0000 7fc1e2eee700 10 monclient: get_monmap_and_config 2023-09-30T00:47:52.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.579+0000 7fc1e2eee700 10 monclient: build_initial_monmap 2023-09-30T00:47:52.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.579+0000 7fc1e2eee700 10 monclient: monmap: 2023-09-30T00:47:52.580 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:52.580 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:52.580 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:52.580228+0000 2023-09-30T00:47:52.580 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:52.580228+0000 2023-09-30T00:47:52.580 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:52.580 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:52.580 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:52.581 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:52.581 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:52.581 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:52.581 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:52.581 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:52.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 1 Processor -- start 2023-09-30T00:47:52.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 1 -- start start 2023-09-30T00:47:52.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 10 monclient: init 2023-09-30T00:47:52.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:52.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-09-30T00:47:52.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc1dc0a5710 0x7fc1dc0b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:52.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 10 monclient(hunting): picked mon.noname-a con 0x7fc1dc0a5710 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:52.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1dc0a7630 0x7fc1dc0a7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:52.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 10 monclient(hunting): picked mon.noname-b con 0x7fc1dc0a7630 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:52.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc1dc0a7fe0 0x7fc1dc0af3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:52.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 10 monclient(hunting): picked mon.noname-c con 0x7fc1dc0a7fe0 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:52.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1dc0b1770 con 0x7fc1dc0a5710 2023-09-30T00:47:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1dc0a5ae0 con 0x7fc1dc0a7fe0 2023-09-30T00:47:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1dc0b1040 con 0x7fc1dc0a7630 2023-09-30T00:47:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.580+0000 7fc1e2eee700 10 monclient(hunting): authenticate will time out at 5333.881936s 2023-09-30T00:47:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.581+0000 7fc1e0c8a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc1dc0a5710 0x7fc1dc0b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.582+0000 7fc1e0c8a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc1dc0a5710 0x7fc1dc0b0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56426/0 (socket says 172.21.15.40:56426) 2023-09-30T00:47:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.582+0000 7fc1e0c8a700 1 -- 172.21.15.40:0/3797588124 learned_addr learned my addr 172.21.15.40:0/3797588124 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.582+0000 7fc1e0c8a700 10 monclient(hunting): get_auth_request con 0x7fc1dc0a5710 auth_method 0 2023-09-30T00:47:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.582+0000 7fc1e0c8a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:52.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.582+0000 7fc1e0c8a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:52.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.582+0000 7fc1e0c8a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:52.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.582+0000 7fc1e0c8a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:52.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.582+0000 7fc1e0c8a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:52.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.582+0000 7fc1e0c8a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:52.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.585+0000 7fc1e0c8a700 10 monclient(hunting): handle_auth_done global_id 4254 payload 274 2023-09-30T00:47:52.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.585+0000 7fc1e0c8a700 1 -- 172.21.15.40:0/3797588124 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1dc0a7630 msgr2=0x7fc1dc0a7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:52.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.585+0000 7fc1e0c8a700 1 --2- 172.21.15.40:0/3797588124 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1dc0a7630 0x7fc1dc0a7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.585+0000 7fc1e0c8a700 1 -- 172.21.15.40:0/3797588124 >> v2:172.21.15.135:3300/0 conn(0x7fc1dc0a7fe0 msgr2=0x7fc1dc0af3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:52.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.585+0000 7fc1e0c8a700 1 --2- 172.21.15.40:0/3797588124 >> v2:172.21.15.135:3300/0 conn(0x7fc1dc0a7fe0 0x7fc1dc0af3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.585+0000 7fc1e0c8a700 10 monclient: _finish_hunting 0 2023-09-30T00:47:52.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.585+0000 7fc1e0c8a700 1 monclient: found mon.noname-a 2023-09-30T00:47:52.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.585+0000 7fc1e0c8a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:52.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.586+0000 7fc1e0c8a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:52.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.586+0000 7fc1e0c8a700 1 -- 172.21.15.40:0/3797588124 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1dc0b1440 con 0x7fc1dc0a5710 2023-09-30T00:47:52.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.589+0000 7fc1e0c8a700 1 --2- 172.21.15.40:0/3797588124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1dc0a5710 0x7fc1dc0b0b00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc0abac0 tx=0x7fc1d0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1e1ed4e87c4a24 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:52.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.594+0000 7fc1db7fe700 1 -- 172.21.15.40:0/3797588124 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d000b860 con 0x7fc1dc0a5710 2023-09-30T00:47:52.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.594+0000 7fc1db7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:52.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.594+0000 7fc1db7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:52.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.594+0000 7fc1db7fe700 10 monclient: dump: 2023-09-30T00:47:52.597 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:52.597 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:52.597 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:52.597 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:52.597 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:52.597 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:52.597 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:52.597 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:52.597 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:52.597 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:52.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.594+0000 7fc1db7fe700 10 monclient: _finish_auth 0 2023-09-30T00:47:52.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.594+0000 7fc1db7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:52.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.594+0000 7fc1db7fe700 1 -- 172.21.15.40:0/3797588124 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d000b9c0 con 0x7fc1dc0a5710 2023-09-30T00:47:52.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.594+0000 7fc1e2eee700 5 monclient: authenticate success, global_id 4254 2023-09-30T00:47:52.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.594+0000 7fc1e2eee700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:52.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.594+0000 7fc1db7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:52.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.594+0000 7fc1e2eee700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:52.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.594+0000 7fc1db7fe700 1 -- 172.21.15.40:0/3797588124 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d000bd60 con 0x7fc1dc0a5710 2023-09-30T00:47:52.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.594+0000 7fc1db7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:52.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.594+0000 7fc1e2eee700 4 set_mon_vals no callback set 2023-09-30T00:47:52.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.595+0000 7fc1db7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:52.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.595+0000 7fc1db7fe700 10 monclient: dump: 2023-09-30T00:47:52.599 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:52.599 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:52.599 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:52.599 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:52.599 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:52.599 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:52.599 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:52.599 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:52.600 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:52.600 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:52.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.595+0000 7fc1e2eee700 10 monclient: shutdown 2023-09-30T00:47:52.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.595+0000 7fc1e2eee700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:52.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.595+0000 7fc1e2eee700 1 -- 172.21.15.40:0/3797588124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1dc0a5710 msgr2=0x7fc1dc0b0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:52.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.595+0000 7fc1e2eee700 1 --2- 172.21.15.40:0/3797588124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1dc0a5710 0x7fc1dc0b0b00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc0abac0 tx=0x7fc1d0009250 comp rx=0 tx=0).stop 2023-09-30T00:47:52.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.595+0000 7fc1e2eee700 1 -- 172.21.15.40:0/3797588124 shutdown_connections 2023-09-30T00:47:52.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.595+0000 7fc1e2eee700 1 --2- 172.21.15.40:0/3797588124 >> v2:172.21.15.135:3300/0 conn(0x7fc1dc0a7fe0 0x7fc1dc0af3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.595+0000 7fc1e2eee700 1 --2- 172.21.15.40:0/3797588124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1dc0a5710 0x7fc1dc0b0b00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.595+0000 7fc1e2eee700 1 --2- 172.21.15.40:0/3797588124 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1dc0a7630 0x7fc1dc0a7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.595+0000 7fc1e2eee700 1 -- 172.21.15.40:0/3797588124 >> 172.21.15.40:0/3797588124 conn(0x7fc1dc069060 msgr2=0x7fc1dc0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:52.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.595+0000 7fc1e2eee700 1 -- 172.21.15.40:0/3797588124 shutdown_connections 2023-09-30T00:47:52.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.595+0000 7fc1e2eee700 1 -- 172.21.15.40:0/3797588124 wait complete. 2023-09-30T00:47:52.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.596+0000 7fc1e2eee700 10 monclient: build_initial_monmap 2023-09-30T00:47:52.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.596+0000 7fc1e2eee700 10 monclient: monmap: 2023-09-30T00:47:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:52.603 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:52.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.597+0000 7fc1e2eee700 1 librados: starting msgr at 2023-09-30T00:47:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.597+0000 7fc1e2eee700 1 librados: starting objecter 2023-09-30T00:47:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.597+0000 7fc1e2eee700 1 Processor -- start 2023-09-30T00:47:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.597+0000 7fc1e2eee700 1 -- start start 2023-09-30T00:47:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.597+0000 7fc1e2eee700 1 librados: setting wanted keys 2023-09-30T00:47:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.597+0000 7fc1e2eee700 1 librados: calling monclient init 2023-09-30T00:47:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.597+0000 7fc1e2eee700 10 monclient: init 2023-09-30T00:47:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:47:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1dc0a5710 0x7fc1dc13b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 10 monclient(hunting): picked mon.noname-b con 0x7fc1dc0a5710 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1dc0a7630 0x7fc1dc13bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:52.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 10 monclient(hunting): picked mon.noname-c con 0x7fc1dc0a7630 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:52.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1dc0a7fe0 0x7fc1dc13c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:52.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 10 monclient(hunting): picked mon.noname-a con 0x7fc1dc0a7fe0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:52.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:52.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1dc13eca0 con 0x7fc1dc0a7fe0 2023-09-30T00:47:52.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:52.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1dc13f130 con 0x7fc1dc0a5710 2023-09-30T00:47:52.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:52.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1dc13ca50 con 0x7fc1dc0a7630 2023-09-30T00:47:52.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:52.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.598+0000 7fc1e2eee700 10 monclient(hunting): authenticate will time out at 5333.899604s 2023-09-30T00:47:52.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.599+0000 7fc1e148b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1dc0a7fe0 0x7fc1dc13c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:52.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.599+0000 7fc1e148b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1dc0a7fe0 0x7fc1dc13c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56428/0 (socket says 172.21.15.40:56428) 2023-09-30T00:47:52.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.599+0000 7fc1e148b700 1 -- 172.21.15.40:0/1428016429 learned_addr learned my addr 172.21.15.40:0/1428016429 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:52.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.599+0000 7fc1e148b700 10 monclient(hunting): get_auth_request con 0x7fc1dc0a7fe0 auth_method 0 2023-09-30T00:47:52.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.599+0000 7fc1e148b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:52.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.599+0000 7fc1e148b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:52.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.599+0000 7fc1e148b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:52.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.600+0000 7fc1e148b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:52.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.600+0000 7fc1e148b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:52.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.600+0000 7fc1e148b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:52.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.604+0000 7fc1e148b700 10 monclient(hunting): handle_auth_done global_id 4257 payload 824 2023-09-30T00:47:52.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.604+0000 7fc1e148b700 1 -- 172.21.15.40:0/1428016429 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1dc0a7630 msgr2=0x7fc1dc13bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:52.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.604+0000 7fc1e148b700 1 --2- 172.21.15.40:0/1428016429 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1dc0a7630 0x7fc1dc13bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.604+0000 7fc1e148b700 1 -- 172.21.15.40:0/1428016429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1dc0a5710 msgr2=0x7fc1dc13b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:52.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.604+0000 7fc1e148b700 1 --2- 172.21.15.40:0/1428016429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1dc0a5710 0x7fc1dc13b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.604+0000 7fc1e148b700 10 monclient: _finish_hunting 0 2023-09-30T00:47:52.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.604+0000 7fc1e148b700 1 monclient: found mon.noname-a 2023-09-30T00:47:52.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.604+0000 7fc1e148b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:52.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.604+0000 7fc1e148b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:52.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.604+0000 7fc1e148b700 1 -- 172.21.15.40:0/1428016429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d00094f0 con 0x7fc1dc0a7fe0 2023-09-30T00:47:52.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.607+0000 7fc1e148b700 1 --2- 172.21.15.40:0/1428016429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1dc0a7fe0 0x7fc1dc13c510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc1d400bb80 tx=0x7fc1d40097c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:52.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.607+0000 7fc1d8ff9700 10 client.?.objecter ms_handle_connect 0x7fc1dc0a7fe0 2023-09-30T00:47:52.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.607+0000 7fc1d8ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:52.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1d8ff9700 1 -- 172.21.15.40:0/1428016429 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d400c900 con 0x7fc1dc0a7fe0 2023-09-30T00:47:52.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1d8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:52.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1d8ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:52.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1d8ff9700 10 monclient: dump: 2023-09-30T00:47:52.613 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:52.613 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:52.613 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:52.614 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:52.614 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:52.614 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:52.614 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:52.614 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:52.614 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:52.614 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:52.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1d8ff9700 10 monclient: _finish_auth 0 2023-09-30T00:47:52.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1d8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:52.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1d8ff9700 1 -- 172.21.15.40:0/1428016429 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d400ca80 con 0x7fc1dc0a7fe0 2023-09-30T00:47:52.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1d8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:52.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1d8ff9700 1 -- 172.21.15.40:0/1428016429 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d400ce40 con 0x7fc1dc0a7fe0 2023-09-30T00:47:52.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1e2eee700 5 monclient: authenticate success, global_id 4257 2023-09-30T00:47:52.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1d8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:52.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1d8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:52.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1d8ff9700 10 monclient: dump: 2023-09-30T00:47:52.615 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:52.615 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:52.615 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:52.615 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:52.615 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:52.615 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:52.615 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:52.616 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:52.616 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:52.616 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:52.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1e2eee700 10 monclient: _renew_subs 2023-09-30T00:47:52.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1e2eee700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:52.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1e2eee700 1 -- 172.21.15.40:0/1428016429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1dc13e8f0 con 0x7fc1dc0a7fe0 2023-09-30T00:47:52.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1daffd700 4 set_mon_vals no callback set 2023-09-30T00:47:52.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1e2eee700 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:52.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1e2eee700 10 monclient: _renew_subs 2023-09-30T00:47:52.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1e2eee700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:52.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1e2eee700 1 -- 172.21.15.40:0/1428016429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1dc0a8710 con 0x7fc1dc0a7fe0 2023-09-30T00:47:52.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.611+0000 7fc1e2eee700 1 librados: init done 2023-09-30T00:47:52.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.612+0000 7fc1e2eee700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:52.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.612+0000 7fc1e2eee700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:52.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.613+0000 7fc1e2eee700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:52.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.613+0000 7fc1e2eee700 1 -- 172.21.15.40:0/1428016429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1dc064a00 con 0x7fc1dc0a7fe0 2023-09-30T00:47:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.641+0000 7fc1d8ff9700 1 -- 172.21.15.40:0/1428016429 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fc1d4014620 con 0x7fc1dc0a7fe0 2023-09-30T00:47:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.642+0000 7fc1d8ff9700 1 --2- 172.21.15.40:0/1428016429 >> v2:172.21.15.135:6800/116426 conn(0x7fc1c0068960 0x7fc1c006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.642+0000 7fc1d8ff9700 1 -- 172.21.15.40:0/1428016429 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4075+0+0 (secure 0 0 0) 0x7fc1d40846b0 con 0x7fc1dc0a7fe0 2023-09-30T00:47:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.642+0000 7fc1d8ff9700 10 client.4257.objecter ms_dispatch 0x7fc1dc001550 osd_map(22..22 src has 1..22) v4 2023-09-30T00:47:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.642+0000 7fc1d8ff9700 3 client.4257.objecter handle_osd_map got epochs [22,22] > 0 2023-09-30T00:47:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.642+0000 7fc1d8ff9700 3 client.4257.objecter handle_osd_map decoding full epoch 22 2023-09-30T00:47:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.642+0000 7fc1d8ff9700 20 client.4257.objecter dump_active .. 0 homeless 2023-09-30T00:47:52.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.642+0000 7fc1e0c8a700 1 --2- 172.21.15.40:0/1428016429 >> v2:172.21.15.135:6800/116426 conn(0x7fc1c0068960 0x7fc1c006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:52.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.642+0000 7fc1e0c8a700 10 monclient: get_auth_request con 0x7fc1c0068960 auth_method 0 2023-09-30T00:47:52.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.643+0000 7fc1d8ff9700 10 client.4257.objecter ms_handle_connect 0x7fc1c0068960 2023-09-30T00:47:52.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.643+0000 7fc1e0c8a700 1 --2- 172.21.15.40:0/1428016429 >> v2:172.21.15.135:6800/116426 conn(0x7fc1c0068960 0x7fc1c006ae10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc0abac0 tx=0x7fc1d0017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:52.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.953+0000 7fc1d8ff9700 1 -- 172.21.15.40:0/1428016429 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc1d40520e0 con 0x7fc1dc0a7fe0 2023-09-30T00:47:52.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.953+0000 7fc1d8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:52.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:52.953+0000 7fc1d8ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:53.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.263+0000 7fc1e2eee700 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T00:47:53.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.263+0000 7fc1e2eee700 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T00:47:53.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.263+0000 7fc1e2eee700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:53.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.263+0000 7fc1e2eee700 1 -- 172.21.15.40:0/1428016429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc1dc0a7210 con 0x7fc1dc0a7fe0 2023-09-30T00:47:53.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.576+0000 7fc1d8ff9700 1 -- 172.21.15.40:0/1428016429 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v23) v1 ==== 173+0+0 (secure 0 0 0) 0x7fc1d4055700 con 0x7fc1dc0a7fe0 2023-09-30T00:47:53.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.576+0000 7fc1d8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T00:47:53.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.576+0000 7fc1d8ff9700 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-09-30T00:47:53.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 10 librados: watch_flush enter 2023-09-30T00:47:53.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 10 librados: watch_flush exit 2023-09-30T00:47:53.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 20 client.4257.objecter shutdown clearing up homeless session... 2023-09-30T00:47:53.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 10 client.4257.objecter successfully canceled tick 2023-09-30T00:47:53.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 1 -- 172.21.15.40:0/1428016429 >> v2:172.21.15.135:6800/116426 conn(0x7fc1c0068960 msgr2=0x7fc1c006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:53.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 1 --2- 172.21.15.40:0/1428016429 >> v2:172.21.15.135:6800/116426 conn(0x7fc1c0068960 0x7fc1c006ae10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc0abac0 tx=0x7fc1d0017000 comp rx=0 tx=0).stop 2023-09-30T00:47:53.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 10 monclient: shutdown 2023-09-30T00:47:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 1 -- 172.21.15.40:0/1428016429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1dc0a7fe0 msgr2=0x7fc1dc13c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 1 --2- 172.21.15.40:0/1428016429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1dc0a7fe0 0x7fc1dc13c510 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc1d400bb80 tx=0x7fc1d40097c0 comp rx=0 tx=0).stop 2023-09-30T00:47:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 1 -- 172.21.15.40:0/1428016429 shutdown_connections 2023-09-30T00:47:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 1 --2- 172.21.15.40:0/1428016429 >> v2:172.21.15.135:6800/116426 conn(0x7fc1c0068960 0x7fc1c006ae10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 1 --2- 172.21.15.40:0/1428016429 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc1dc0a5710 0x7fc1dc13b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 1 --2- 172.21.15.40:0/1428016429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1dc0a7fe0 0x7fc1dc13c510 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 1 --2- 172.21.15.40:0/1428016429 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc1dc0a7630 0x7fc1dc13bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 1 -- 172.21.15.40:0/1428016429 >> 172.21.15.40:0/1428016429 conn(0x7fc1dc069060 msgr2=0x7fc1dc0b2880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 1 -- 172.21.15.40:0/1428016429 shutdown_connections 2023-09-30T00:47:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 1 -- 172.21.15.40:0/1428016429 wait complete. 2023-09-30T00:47:53.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.579+0000 7fc1e2eee700 1 librados: shutdown 2023-09-30T00:47:53.582 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-30T00:47:53.596 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-30T00:47:53.596 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-30T00:47:53.596 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-30T00:47:53.596 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-30T00:47:53.597 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-30T00:47:53.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.738+0000 7f55c97fd700 10 monclient: get_monmap_and_config 2023-09-30T00:47:53.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.738+0000 7f55c97fd700 10 monclient: build_initial_monmap 2023-09-30T00:47:53.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.738+0000 7f55c97fd700 10 monclient: monmap: 2023-09-30T00:47:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:53.739226+0000 2023-09-30T00:47:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:53.739226+0000 2023-09-30T00:47:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:53.742 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:53.743 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.738+0000 7f55c97fd700 1 Processor -- start 2023-09-30T00:47:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 1 -- start start 2023-09-30T00:47:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 10 monclient: init 2023-09-30T00:47:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2023-09-30T00:47:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f55c40ae830 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40ad6b0 0x7f55c4141a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 10 monclient(hunting): picked mon.noname-b con 0x7f55c40ad6b0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f55c4141f70 0x7f55c41423b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 10 monclient(hunting): picked mon.noname-c con 0x7f55c4141f70 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:53.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c406fc90 con 0x7f55c40ae830 2023-09-30T00:47:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55c41428f0 con 0x7f55c4141f70 2023-09-30T00:47:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c40a60f0 con 0x7f55c40ad6b0 2023-09-30T00:47:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.739+0000 7f55c97fd700 10 monclient(hunting): authenticate will time out at 5335.040839s 2023-09-30T00:47:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.740+0000 7f55c2ffd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f55c40ae830 0x7f55c40acfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43266/0 (socket says 172.21.15.40:43266) 2023-09-30T00:47:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.740+0000 7f55c2ffd700 1 -- 172.21.15.40:0/1205092600 learned_addr learned my addr 172.21.15.40:0/1205092600 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.740+0000 7f55c27fc700 1 --2- 172.21.15.40:0/1205092600 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40ad6b0 0x7f55c4141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.740+0000 7f55c37fe700 1 --2- 172.21.15.40:0/1205092600 >> v2:172.21.15.135:3300/0 conn(0x7f55c4141f70 0x7f55c41423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.741+0000 7f55c37fe700 10 monclient(hunting): get_auth_request con 0x7f55c4141f70 auth_method 0 2023-09-30T00:47:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.741+0000 7f55c37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.741+0000 7f55c37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.741+0000 7f55c37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.741+0000 7f55c27fc700 10 monclient(hunting): get_auth_request con 0x7f55c40ad6b0 auth_method 0 2023-09-30T00:47:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.741+0000 7f55c27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.741+0000 7f55c27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.741+0000 7f55c27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:53.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.741+0000 7f55c37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:53.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.741+0000 7f55c37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:53.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.741+0000 7f55c37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:53.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.741+0000 7f55c27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:53.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.741+0000 7f55c27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.741+0000 7f55c27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.744+0000 7f55c27fc700 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2023-09-30T00:47:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.744+0000 7f55c27fc700 1 -- 172.21.15.40:0/1205092600 >> v2:172.21.15.135:3300/0 conn(0x7f55c4141f70 msgr2=0x7f55c41423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.744+0000 7f55c27fc700 1 --2- 172.21.15.40:0/1205092600 >> v2:172.21.15.135:3300/0 conn(0x7f55c4141f70 0x7f55c41423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.744+0000 7f55c27fc700 1 -- 172.21.15.40:0/1205092600 >> v1:172.21.15.40:6789/0 conn(0x7f55c40ae830 legacy=0x7f55c40acfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.744+0000 7f55c27fc700 10 monclient: _finish_hunting 0 2023-09-30T00:47:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.744+0000 7f55c27fc700 1 monclient: found mon.noname-b 2023-09-30T00:47:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.744+0000 7f55c27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.744+0000 7f55c27fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:47:53.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.744+0000 7f55c27fc700 1 -- 172.21.15.40:0/1205092600 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c40aec00 con 0x7f55c40ad6b0 2023-09-30T00:47:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.746+0000 7f55c27fc700 1 --2- 172.21.15.40:0/1205092600 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40ad6b0 0x7f55c4141a30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f55b80098e0 tx=0x7f55b80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c593827a186b806b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c1ffb700 1 -- 172.21.15.40:0/1205092600 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b800b640 con 0x7f55c40ad6b0 2023-09-30T00:47:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c1ffb700 10 monclient: dump: 2023-09-30T00:47:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:53.753 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:53.754 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c1ffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c97fd700 5 monclient: authenticate success, global_id 4223 2023-09-30T00:47:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c1ffb700 1 -- 172.21.15.40:0/1205092600 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b800b7a0 con 0x7f55c40ad6b0 2023-09-30T00:47:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:53.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c97fd700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c97fd700 4 set_mon_vals no callback set 2023-09-30T00:47:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c1ffb700 1 -- 172.21.15.40:0/1205092600 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b800bb40 con 0x7f55c40ad6b0 2023-09-30T00:47:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.752+0000 7f55c1ffb700 10 monclient: dump: 2023-09-30T00:47:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:53.755 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:53.756 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:53.756 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:53.756 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:53.756 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:53.756 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:53.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.753+0000 7f55c97fd700 10 monclient: shutdown 2023-09-30T00:47:53.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.753+0000 7f55c97fd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:53.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.753+0000 7f55c97fd700 1 -- 172.21.15.40:0/1205092600 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40ad6b0 msgr2=0x7f55c4141a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:53.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.753+0000 7f55c97fd700 1 --2- 172.21.15.40:0/1205092600 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40ad6b0 0x7f55c4141a30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f55b80098e0 tx=0x7f55b80091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:53.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.753+0000 7f55c97fd700 1 -- 172.21.15.40:0/1205092600 shutdown_connections 2023-09-30T00:47:53.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.753+0000 7f55c97fd700 1 --2- 172.21.15.40:0/1205092600 >> v2:172.21.15.135:3300/0 conn(0x7f55c4141f70 0x7f55c41423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:53.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.753+0000 7f55c97fd700 1 --2- 172.21.15.40:0/1205092600 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40ad6b0 0x7f55c4141a30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:53.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.753+0000 7f55c97fd700 1 -- 172.21.15.40:0/1205092600 >> 172.21.15.40:0/1205092600 conn(0x7f55c4068920 msgr2=0x7f55c4068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:53.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.754+0000 7f55c97fd700 1 -- 172.21.15.40:0/1205092600 shutdown_connections 2023-09-30T00:47:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.754+0000 7f55c97fd700 1 -- 172.21.15.40:0/1205092600 wait complete. 2023-09-30T00:47:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.755+0000 7f55c97fd700 10 monclient: build_initial_monmap 2023-09-30T00:47:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.755+0000 7f55c97fd700 10 monclient: monmap: 2023-09-30T00:47:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:53.761 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:53.762 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.755+0000 7f55c97fd700 1 librados: starting msgr at 2023-09-30T00:47:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.755+0000 7f55c97fd700 1 librados: starting objecter 2023-09-30T00:47:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.755+0000 7f55c97fd700 1 Processor -- start 2023-09-30T00:47:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.755+0000 7f55c97fd700 1 -- start start 2023-09-30T00:47:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.755+0000 7f55c97fd700 1 librados: setting wanted keys 2023-09-30T00:47:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.755+0000 7f55c97fd700 1 librados: calling monclient init 2023-09-30T00:47:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.755+0000 7f55c97fd700 10 monclient: init 2023-09-30T00:47:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:47:53.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55c40ad6b0 0x7f55c40ab120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 10 monclient(hunting): picked mon.noname-b con 0x7f55c40ad6b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40ae830 0x7f55c411dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 10 monclient(hunting): picked mon.noname-c con 0x7f55c40ae830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55c4141f70 0x7f55c413f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 10 monclient(hunting): picked mon.noname-a con 0x7f55c4141f70 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c411ef30 con 0x7f55c4141f70 2023-09-30T00:47:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c40a8840 con 0x7f55c40ad6b0 2023-09-30T00:47:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c411f1a0 con 0x7f55c40ae830 2023-09-30T00:47:53.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.756+0000 7f55c97fd700 10 monclient(hunting): authenticate will time out at 5335.057601s 2023-09-30T00:47:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c27fc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40ae830 0x7f55c411dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c2ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55c40ad6b0 0x7f55c40ab120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c37fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55c4141f70 0x7f55c413f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c2ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55c40ad6b0 0x7f55c40ab120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43068/0 (socket says 172.21.15.40:43068) 2023-09-30T00:47:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c2ffd700 1 -- 172.21.15.40:0/1689295805 learned_addr learned my addr 172.21.15.40:0/1689295805 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c2ffd700 10 monclient(hunting): get_auth_request con 0x7f55c40ad6b0 auth_method 0 2023-09-30T00:47:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c27fc700 10 monclient(hunting): get_auth_request con 0x7f55c40ae830 auth_method 0 2023-09-30T00:47:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:53.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:53.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c37fe700 10 monclient(hunting): get_auth_request con 0x7f55c4141f70 auth_method 0 2023-09-30T00:47:53.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:53.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:53.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.757+0000 7f55c37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:53.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.758+0000 7f55c2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:53.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.758+0000 7f55c2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:53.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.758+0000 7f55c2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:53.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.758+0000 7f55c27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:53.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.758+0000 7f55c27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:53.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.758+0000 7f55c27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:53.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.758+0000 7f55c37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:53.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.758+0000 7f55c37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:53.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.758+0000 7f55c37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:53.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.761+0000 7f55c27fc700 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2023-09-30T00:47:53.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.762+0000 7f55c27fc700 1 -- 172.21.15.40:0/1689295805 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55c40ad6b0 msgr2=0x7f55c40ab120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:53.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.762+0000 7f55c27fc700 1 --2- 172.21.15.40:0/1689295805 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55c40ad6b0 0x7f55c40ab120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:53.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.762+0000 7f55c27fc700 1 -- 172.21.15.40:0/1689295805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55c4141f70 msgr2=0x7f55c413f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:53.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.762+0000 7f55c27fc700 1 --2- 172.21.15.40:0/1689295805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55c4141f70 0x7f55c413f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:53.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.762+0000 7f55c27fc700 10 monclient: _finish_hunting 0 2023-09-30T00:47:53.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.762+0000 7f55c27fc700 1 monclient: found mon.noname-c 2023-09-30T00:47:53.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.762+0000 7f55c27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:53.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.762+0000 7f55c27fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:47:53.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.762+0000 7f55c27fc700 1 -- 172.21.15.40:0/1689295805 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b8009420 con 0x7f55c40ae830 2023-09-30T00:47:53.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.765+0000 7f55c27fc700 1 --2- 172.21.15.40:0/1689295805 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40ae830 0x7f55c411dcb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f55b8016000 tx=0x7f55b8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:53.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.765+0000 7f55aaffd700 10 client.?.objecter ms_handle_connect 0x7f55c40ae830 2023-09-30T00:47:53.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.765+0000 7f55aaffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.772+0000 7f55aaffd700 1 -- 172.21.15.40:0/1689295805 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b801bb70 con 0x7f55c40ae830 2023-09-30T00:47:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.772+0000 7f55aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.772+0000 7f55aaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.772+0000 7f55aaffd700 10 monclient: dump: 2023-09-30T00:47:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:53.775 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.772+0000 7f55aaffd700 10 monclient: _finish_auth 0 2023-09-30T00:47:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.772+0000 7f55aaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.772+0000 7f55aaffd700 1 -- 172.21.15.40:0/1689295805 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b801bcd0 con 0x7f55c40ae830 2023-09-30T00:47:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.772+0000 7f55aaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.773+0000 7f55aaffd700 1 -- 172.21.15.40:0/1689295805 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b8011670 con 0x7f55c40ae830 2023-09-30T00:47:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.773+0000 7f55c97fd700 5 monclient: authenticate success, global_id 4226 2023-09-30T00:47:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.773+0000 7f55c97fd700 10 monclient: _renew_subs 2023-09-30T00:47:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.773+0000 7f55c97fd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.773+0000 7f55c97fd700 1 -- 172.21.15.40:0/1689295805 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c411e1f0 con 0x7f55c40ae830 2023-09-30T00:47:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.773+0000 7f55aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.773+0000 7f55aaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.773+0000 7f55aaffd700 10 monclient: dump: 2023-09-30T00:47:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:53.777 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.773+0000 7f55c17fa700 4 set_mon_vals no callback set 2023-09-30T00:47:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.773+0000 7f55c97fd700 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.773+0000 7f55c97fd700 10 monclient: _renew_subs 2023-09-30T00:47:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.773+0000 7f55c97fd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.773+0000 7f55c97fd700 1 -- 172.21.15.40:0/1689295805 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c4140020 con 0x7f55c40ae830 2023-09-30T00:47:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.773+0000 7f55c97fd700 1 librados: init done 2023-09-30T00:47:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.774+0000 7f55c97fd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.774+0000 7f55c97fd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.774+0000 7f55c97fd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.774+0000 7f55c97fd700 1 -- 172.21.15.40:0/1689295805 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c40a8a00 con 0x7f55c40ae830 2023-09-30T00:47:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.969+0000 7f55aaffd700 1 -- 172.21.15.40:0/1689295805 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f55b8011a20 con 0x7f55c40ae830 2023-09-30T00:47:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.969+0000 7f55aaffd700 1 --2- 172.21.15.40:0/1689295805 >> v2:172.21.15.135:6800/116426 conn(0x7f55b00689e0 0x7f55b006ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.969+0000 7f55aaffd700 1 -- 172.21.15.40:0/1689295805 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f55b80856e0 con 0x7f55c40ae830 2023-09-30T00:47:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.969+0000 7f55aaffd700 10 client.4226.objecter ms_dispatch 0x7f55c4001550 osd_map(23..23 src has 1..23) v4 2023-09-30T00:47:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.969+0000 7f55aaffd700 3 client.4226.objecter handle_osd_map got epochs [23,23] > 0 2023-09-30T00:47:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.969+0000 7f55aaffd700 3 client.4226.objecter handle_osd_map decoding full epoch 23 2023-09-30T00:47:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.970+0000 7f55aaffd700 20 client.4226.objecter dump_active .. 0 homeless 2023-09-30T00:47:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.970+0000 7f55c2ffd700 1 --2- 172.21.15.40:0/1689295805 >> v2:172.21.15.135:6800/116426 conn(0x7f55b00689e0 0x7f55b006ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:53.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.970+0000 7f55c2ffd700 10 monclient: get_auth_request con 0x7f55b00689e0 auth_method 0 2023-09-30T00:47:53.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.970+0000 7f55aaffd700 10 client.4226.objecter ms_handle_connect 0x7f55b00689e0 2023-09-30T00:47:53.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:53.970+0000 7f55c2ffd700 1 --2- 172.21.15.40:0/1689295805 >> v2:172.21.15.135:6800/116426 conn(0x7f55b00689e0 0x7f55b006ae90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f55b4007130 tx=0x7f55b4007f70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:54.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.133+0000 7f55aaffd700 1 -- 172.21.15.40:0/1689295805 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f55b8053000 con 0x7f55c40ae830 2023-09-30T00:47:54.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.133+0000 7f55aaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:54.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.133+0000 7f55aaffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:54.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.417+0000 7f55c97fd700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-09-30T00:47:54.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.417+0000 7f55c97fd700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-09-30T00:47:54.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.417+0000 7f55c97fd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:54.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.417+0000 7f55c97fd700 1 -- 172.21.15.40:0/1689295805 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f55c4064a00 con 0x7f55c40ae830 2023-09-30T00:47:54.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.610+0000 7f55aaffd700 1 -- 172.21.15.40:0/1689295805 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+163650 (secure 0 0 0) 0x7f55b8056620 con 0x7f55c40ae830 2023-09-30T00:47:54.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.610+0000 7f55aaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-09-30T00:47:54.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.610+0000 7f55aaffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:54.614 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:47:54.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 10 librados: watch_flush enter 2023-09-30T00:47:54.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 10 librados: watch_flush exit 2023-09-30T00:47:54.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 20 client.4226.objecter shutdown clearing up homeless session... 2023-09-30T00:47:54.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 10 client.4226.objecter successfully canceled tick 2023-09-30T00:47:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 1 -- 172.21.15.40:0/1689295805 >> v2:172.21.15.135:6800/116426 conn(0x7f55b00689e0 msgr2=0x7f55b006ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 1 --2- 172.21.15.40:0/1689295805 >> v2:172.21.15.135:6800/116426 conn(0x7f55b00689e0 0x7f55b006ae90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f55b4007130 tx=0x7f55b4007f70 comp rx=0 tx=0).stop 2023-09-30T00:47:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 10 monclient: shutdown 2023-09-30T00:47:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 1 -- 172.21.15.40:0/1689295805 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40ae830 msgr2=0x7f55c411dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 1 --2- 172.21.15.40:0/1689295805 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40ae830 0x7f55c411dcb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f55b8016000 tx=0x7f55b8017000 comp rx=0 tx=0).stop 2023-09-30T00:47:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 1 -- 172.21.15.40:0/1689295805 shutdown_connections 2023-09-30T00:47:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 1 --2- 172.21.15.40:0/1689295805 >> v2:172.21.15.135:6800/116426 conn(0x7f55b00689e0 0x7f55b006ae90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 1 --2- 172.21.15.40:0/1689295805 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55c40ad6b0 0x7f55c40ab120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 1 --2- 172.21.15.40:0/1689295805 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55c4141f70 0x7f55c413f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 1 --2- 172.21.15.40:0/1689295805 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40ae830 0x7f55c411dcb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 1 -- 172.21.15.40:0/1689295805 >> 172.21.15.40:0/1689295805 conn(0x7f55c4068920 msgr2=0x7f55c4067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:54.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 1 -- 172.21.15.40:0/1689295805 shutdown_connections 2023-09-30T00:47:54.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.613+0000 7f55c97fd700 1 -- 172.21.15.40:0/1689295805 wait complete. 2023-09-30T00:47:54.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.614+0000 7f55c97fd700 1 librados: shutdown 2023-09-30T00:47:54.636 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6800","nonce":116426}]},"active_addr":"172.21.15.135:6800/116426","active_change":"2023-09-30T00:46:53.193132+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4108,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.135:0","nonce":138651422}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.135:0","nonce":333106264}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.135:0","nonce":4281042481}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.135:0","nonce":3723849070}]}]} 2023-09-30T00:47:54.642 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-30T00:47:54.642 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-30T00:47:54.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:47:54.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.817+0000 7fcbb7aa5700 10 monclient: get_monmap_and_config 2023-09-30T00:47:54.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.817+0000 7fcbb7aa5700 10 monclient: build_initial_monmap 2023-09-30T00:47:54.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.817+0000 7fcbb7aa5700 10 monclient: monmap: 2023-09-30T00:47:54.818 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:54.818 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:54.818 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:54.818277+0000 2023-09-30T00:47:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:54.818277+0000 2023-09-30T00:47:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:54.819 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.817+0000 7fcbb7aa5700 1 Processor -- start 2023-09-30T00:47:54.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.818+0000 7fcbb7aa5700 1 -- start start 2023-09-30T00:47:54.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.818+0000 7fcbb7aa5700 10 monclient: init 2023-09-30T00:47:54.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.818+0000 7fcbb7aa5700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:54.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.818+0000 7fcbb7aa5700 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-09-30T00:47:54.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.818+0000 7fcbb7aa5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcbb00b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:54.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.818+0000 7fcbb7aa5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcbb00b1830 0x7fcbb00a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:54.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.818+0000 7fcbb7aa5700 10 monclient(hunting): picked mon.noname-a con 0x7fcbb00b1830 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:54.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.818+0000 7fcbb7aa5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbb013ac20 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:54.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.818+0000 7fcbb7aa5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb00a5b20 con 0x7fcbb013ac20 2023-09-30T00:47:54.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.818+0000 7fcbb7aa5700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb00a71f0 con 0x7fcbb00b1460 2023-09-30T00:47:54.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.818+0000 7fcbb7aa5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:54.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.818+0000 7fcbb7aa5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbb0141e40 con 0x7fcbb00b1830 2023-09-30T00:47:54.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.818+0000 7fcbb7aa5700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:54.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.818+0000 7fcbb7aa5700 10 monclient(hunting): authenticate will time out at 5336.119880s 2023-09-30T00:47:54.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.824+0000 7fcbb5841700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fcbb00b1460 0x7fcbb0141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:53964/0 (socket says 172.21.15.40:53964) 2023-09-30T00:47:54.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.824+0000 7fcbb5841700 1 -- 172.21.15.40:0/4294940068 learned_addr learned my addr 172.21.15.40:0/4294940068 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:54.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.836+0000 7fcbb5040700 1 --2- 172.21.15.40:0/4294940068 >> v2:172.21.15.40:3300/0 conn(0x7fcbb00b1830 0x7fcbb00a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:54.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.837+0000 7fcbb5040700 10 monclient(hunting): get_auth_request con 0x7fcbb00b1830 auth_method 0 2023-09-30T00:47:54.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.837+0000 7fcbb5040700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:54.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.837+0000 7fcbb5040700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:54.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.837+0000 7fcbb5040700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:54.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.839+0000 7fcbb5040700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:54.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.839+0000 7fcbb5040700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:54.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.839+0000 7fcbb5040700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:54.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.841+0000 7fcbb483f700 1 -- 172.21.15.40:0/4294940068 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214920194 0 0) 0x7fcbb00a71f0 con 0x7fcbb00b1460 2023-09-30T00:47:54.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.841+0000 7fcbb483f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:54.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.841+0000 7fcbb483f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:54.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.841+0000 7fcbb483f700 10 monclient(hunting): my global_id is 4288 2023-09-30T00:47:54.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.841+0000 7fcbb483f700 1 -- 172.21.15.40:0/4294940068 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb98003570 con 0x7fcbb00b1460 2023-09-30T00:47:54.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.843+0000 7fcbb5040700 10 monclient(hunting): handle_auth_done global_id 4266 payload 274 2023-09-30T00:47:54.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.843+0000 7fcbb5040700 1 -- 172.21.15.40:0/4294940068 >> v1:172.21.15.135:6789/0 conn(0x7fcbb00b1460 legacy=0x7fcbb0141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:54.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.843+0000 7fcbb5040700 1 -- 172.21.15.40:0/4294940068 >> v1:172.21.15.40:6789/0 conn(0x7fcbb013ac20 legacy=0x7fcbb013b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.843+0000 7fcbb5040700 10 monclient: _finish_hunting 0 2023-09-30T00:47:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.843+0000 7fcbb5040700 1 monclient: found mon.noname-a 2023-09-30T00:47:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.843+0000 7fcbb5040700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.843+0000 7fcbb5040700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.843+0000 7fcbb5040700 1 -- 172.21.15.40:0/4294940068 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb00a74d0 con 0x7fcbb00b1830 2023-09-30T00:47:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.847+0000 7fcbb5040700 1 --2- 172.21.15.40:0/4294940068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbb00b1830 0x7fcbb00a5fb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009910 tx=0x7fcba40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4ac7ebe5eef250 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:54.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb483f700 1 -- 172.21.15.40:0/4294940068 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba4011630 con 0x7fcbb00b1830 2023-09-30T00:47:54.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb483f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:54.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb483f700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:54.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb483f700 10 monclient: dump: 2023-09-30T00:47:54.854 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:54.854 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:54.854 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:54.854 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:54.854 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:54.854 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:54.854 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:54.854 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:54.854 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:54.855 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb483f700 10 monclient: _finish_auth 0 2023-09-30T00:47:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb483f700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb483f700 1 -- 172.21.15.40:0/4294940068 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba4011790 con 0x7fcbb00b1830 2023-09-30T00:47:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb7aa5700 5 monclient: authenticate success, global_id 4266 2023-09-30T00:47:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb7aa5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb483f700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb7aa5700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb483f700 1 -- 172.21.15.40:0/4294940068 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba4011b50 con 0x7fcbb00b1830 2023-09-30T00:47:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb7aa5700 4 set_mon_vals no callback set 2023-09-30T00:47:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb483f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb483f700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.852+0000 7fcbb483f700 10 monclient: dump: 2023-09-30T00:47:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:54.857 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:54.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.853+0000 7fcbb7aa5700 10 monclient: shutdown 2023-09-30T00:47:54.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.853+0000 7fcbb7aa5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:54.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.853+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/4294940068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbb00b1830 msgr2=0x7fcbb00a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:54.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.853+0000 7fcbb7aa5700 1 --2- 172.21.15.40:0/4294940068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbb00b1830 0x7fcbb00a5fb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcba4009910 tx=0x7fcba40091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:54.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.853+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/4294940068 shutdown_connections 2023-09-30T00:47:54.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.853+0000 7fcbb7aa5700 1 --2- 172.21.15.40:0/4294940068 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbb00b1830 0x7fcbb00a5fb0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:54.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.853+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/4294940068 >> 172.21.15.40:0/4294940068 conn(0x7fcbb00631f0 msgr2=0x7fcbb00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:54.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.853+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/4294940068 shutdown_connections 2023-09-30T00:47:54.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.853+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/4294940068 wait complete. 2023-09-30T00:47:54.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.854+0000 7fcbb7aa5700 10 monclient: build_initial_monmap 2023-09-30T00:47:54.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.854+0000 7fcbb7aa5700 10 monclient: monmap: 2023-09-30T00:47:54.861 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:54.861 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:54.861 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:54.861 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:54.861 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:54.861 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:54.861 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:54.861 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:54.861 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:54.861 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:54.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.854+0000 7fcbb7aa5700 1 librados: starting msgr at 2023-09-30T00:47:54.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.854+0000 7fcbb7aa5700 1 librados: starting objecter 2023-09-30T00:47:54.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.854+0000 7fcbb7aa5700 1 Processor -- start 2023-09-30T00:47:54.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 1 -- start start 2023-09-30T00:47:54.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 1 librados: setting wanted keys 2023-09-30T00:47:54.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 1 librados: calling monclient init 2023-09-30T00:47:54.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 10 monclient: init 2023-09-30T00:47:54.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:54.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:47:54.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbb00b1460 0x7fcbb00aa440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:54.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 10 monclient(hunting): picked mon.noname-b con 0x7fcbb00b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:54.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcbb00b1830 0x7fcbb013e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:54.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 10 monclient(hunting): picked mon.noname-c con 0x7fcbb00b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:54.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbb013ac20 0x7fcbb0150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:54.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 10 monclient(hunting): picked mon.noname-a con 0x7fcbb013ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:54.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:54.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb00ab640 con 0x7fcbb013ac20 2023-09-30T00:47:54.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:54.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb00ab8b0 con 0x7fcbb00b1460 2023-09-30T00:47:54.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:54.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb00a9f20 con 0x7fcbb00b1830 2023-09-30T00:47:54.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:54.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.855+0000 7fcbb7aa5700 10 monclient(hunting): authenticate will time out at 5336.156743s 2023-09-30T00:47:54.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.856+0000 7fcbb5841700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbb00b1460 0x7fcbb00aa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:54.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.856+0000 7fcbb5841700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbb00b1460 0x7fcbb00aa440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43084/0 (socket says 172.21.15.40:43084) 2023-09-30T00:47:54.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.856+0000 7fcbb5841700 1 -- 172.21.15.40:0/3961367194 learned_addr learned my addr 172.21.15.40:0/3961367194 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:54.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.856+0000 7fcbb5841700 10 monclient(hunting): get_auth_request con 0x7fcbb00b1460 auth_method 0 2023-09-30T00:47:54.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.856+0000 7fcbb5841700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:54.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.856+0000 7fcbb5841700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:54.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.856+0000 7fcbb5841700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:54.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.857+0000 7fcbb5841700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:54.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.857+0000 7fcbb5841700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:54.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.857+0000 7fcbb5841700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:54.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.857+0000 7fcbb5040700 1 --2- 172.21.15.40:0/3961367194 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcbb00b1830 0x7fcbb013e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:54.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.858+0000 7fcbb5040700 10 monclient(hunting): get_auth_request con 0x7fcbb00b1830 auth_method 0 2023-09-30T00:47:54.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.858+0000 7fcbb5040700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:54.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.858+0000 7fcbb5040700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:54.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.858+0000 7fcbb5040700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:54.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.858+0000 7fcbb5040700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:54.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.858+0000 7fcbb5040700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:54.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.858+0000 7fcbb5040700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:54.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.859+0000 7fcbb6042700 1 --2- 172.21.15.40:0/3961367194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbb013ac20 0x7fcbb0150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:54.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb6042700 10 monclient(hunting): get_auth_request con 0x7fcbb013ac20 auth_method 0 2023-09-30T00:47:54.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb6042700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:54.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb6042700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:54.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb6042700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:54.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb6042700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:54.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb6042700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:54.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb6042700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:54.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb5841700 10 monclient(hunting): handle_auth_done global_id 4291 payload 824 2023-09-30T00:47:54.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb5841700 1 -- 172.21.15.40:0/3961367194 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcbb00b1830 msgr2=0x7fcbb013e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:54.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb5841700 1 --2- 172.21.15.40:0/3961367194 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcbb00b1830 0x7fcbb013e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:54.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb5841700 1 -- 172.21.15.40:0/3961367194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbb013ac20 msgr2=0x7fcbb0150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:54.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb5841700 1 --2- 172.21.15.40:0/3961367194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbb013ac20 0x7fcbb0150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:54.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb5841700 10 monclient: _finish_hunting 0 2023-09-30T00:47:54.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb5841700 1 monclient: found mon.noname-b 2023-09-30T00:47:54.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb5841700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:54.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb5841700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:47:54.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.860+0000 7fcbb5841700 1 -- 172.21.15.40:0/3961367194 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba4009420 con 0x7fcbb00b1460 2023-09-30T00:47:54.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.863+0000 7fcbb5841700 1 --2- 172.21.15.40:0/3961367194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbb00b1460 0x7fcbb00aa440 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcba000cd30 tx=0x7fcba0001b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:54.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.863+0000 7fcbadffb700 10 client.?.objecter ms_handle_connect 0x7fcbb00b1460 2023-09-30T00:47:54.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.863+0000 7fcbadffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:54.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbadffb700 1 -- 172.21.15.40:0/3961367194 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba000ce90 con 0x7fcbb00b1460 2023-09-30T00:47:54.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbadffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:54.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbadffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:54.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbadffb700 10 monclient: dump: 2023-09-30T00:47:54.870 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:54.870 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:54.870 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:54.870 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:54.870 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:54.870 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:54.870 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:54.870 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:54.870 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:54.870 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:54.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbadffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbadffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbadffb700 1 -- 172.21.15.40:0/3961367194 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba000aa60 con 0x7fcbb00b1460 2023-09-30T00:47:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbb7aa5700 5 monclient: authenticate success, global_id 4291 2023-09-30T00:47:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbb7aa5700 10 monclient: _renew_subs 2023-09-30T00:47:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbb7aa5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/3961367194 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb013ee70 con 0x7fcbb00b1460 2023-09-30T00:47:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbadffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbadffb700 1 -- 172.21.15.40:0/3961367194 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba000ae40 con 0x7fcbb00b1460 2023-09-30T00:47:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbadffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbadffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbadffb700 10 monclient: dump: 2023-09-30T00:47:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:54.872 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:54.872 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:54.872 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:54.872 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:54.872 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:54.872 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:54.872 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:54.872 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:54.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbaffff700 4 set_mon_vals no callback set 2023-09-30T00:47:54.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbb7aa5700 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:54.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbb7aa5700 10 monclient: _renew_subs 2023-09-30T00:47:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbb7aa5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/3961367194 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb00a98e0 con 0x7fcbb00b1460 2023-09-30T00:47:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.867+0000 7fcbb7aa5700 1 librados: init done 2023-09-30T00:47:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.868+0000 7fcbb7aa5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.868+0000 7fcbb7aa5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.868+0000 7fcbb7aa5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.868+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/3961367194 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb00aa160 con 0x7fcbb00b1460 2023-09-30T00:47:54.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.893+0000 7fcbadffb700 1 -- 172.21.15.40:0/3961367194 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fcba0007300 con 0x7fcbb00b1460 2023-09-30T00:47:54.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.893+0000 7fcbadffb700 1 --2- 172.21.15.40:0/3961367194 >> v2:172.21.15.135:6800/116426 conn(0x7fcb9c068890 0x7fcb9c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:54.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.894+0000 7fcbb5040700 1 --2- 172.21.15.40:0/3961367194 >> v2:172.21.15.135:6800/116426 conn(0x7fcb9c068890 0x7fcb9c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:54.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.894+0000 7fcbb5040700 10 monclient: get_auth_request con 0x7fcb9c068890 auth_method 0 2023-09-30T00:47:54.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.894+0000 7fcbb5040700 1 --2- 172.21.15.40:0/3961367194 >> v2:172.21.15.135:6800/116426 conn(0x7fcb9c068890 0x7fcb9c06ad40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcba40091e0 tx=0x7fcba4016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:54.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.894+0000 7fcbadffb700 10 client.4291.objecter ms_handle_connect 0x7fcb9c068890 2023-09-30T00:47:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.897+0000 7fcbadffb700 1 -- 172.21.15.40:0/3961367194 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fcba0056870 con 0x7fcbb00b1460 2023-09-30T00:47:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.897+0000 7fcbadffb700 10 client.4291.objecter ms_dispatch 0x7fcbb0001550 osd_map(24..24 src has 1..24) v4 2023-09-30T00:47:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.897+0000 7fcbadffb700 3 client.4291.objecter handle_osd_map got epochs [24,24] > 0 2023-09-30T00:47:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.897+0000 7fcbadffb700 3 client.4291.objecter handle_osd_map decoding full epoch 24 2023-09-30T00:47:54.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:54.897+0000 7fcbadffb700 20 client.4291.objecter dump_active .. 0 homeless 2023-09-30T00:47:55.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.149+0000 7fcbadffb700 1 -- 172.21.15.40:0/3961367194 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcba00524b0 con 0x7fcbb00b1460 2023-09-30T00:47:55.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.149+0000 7fcbadffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:55.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.149+0000 7fcbadffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:55.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.381+0000 7fcbb7aa5700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:55.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.381+0000 7fcbb7aa5700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:55.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.381+0000 7fcbb7aa5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:55.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.381+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/3961367194 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbb00a9d60 con 0x7fcbb00b1460 2023-09-30T00:47:55.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.408+0000 7fcbadffb700 1 -- 172.21.15.40:0/3961367194 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+11948 (secure 0 0 0) 0x7fcba0055ad0 con 0x7fcbb00b1460 2023-09-30T00:47:55.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.408+0000 7fcbadffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:55.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.408+0000 7fcbadffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:55.409 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:47:55.410 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":24,"fsid":"238dba7c-9190-48bd-bb67-322ebc9ecbd0","created":"2023-09-30T00:46:45.822318+0000","modified":"2023-09-30T00:47:54.501141+0000","last_up_change":"2023-09-30T00:47:45.793188+0000","last_in_change":"2023-09-30T00:46:52.657791+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:47:40.523068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:47:51.796955+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"66e1da65-310e-4463-8aa0-431896832e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":117238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":117238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":117238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":117238}]},"public_addr":"172.21.15.40:6804/117238","cluster_addr":"172.21.15.40:6805/117238","heartbeat_back_addr":"172.21.15.40:6807/117238","heartbeat_front_addr":"172.21.15.40:6806/117238","state":["exists","up"]},{"osd":1,"uuid":"d6dd269f-2eda-4ef7-9da1-2ed5b16291b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":117237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6801","nonce":117237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":117237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":117237}]},"public_addr":"172.21.15.40:6800/117237","cluster_addr":"172.21.15.40:6801/117237","heartbeat_back_addr":"172.21.15.40:6803/117237","heartbeat_front_addr":"172.21.15.40:6802/117237","state":["exists","up"]},{"osd":2,"uuid":"8f0de27d-3674-4ff4-8451-4fcc90786c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":117239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":117239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":117239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":117239}]},"public_addr":"172.21.15.40:6808/117239","cluster_addr":"172.21.15.40:6809/117239","heartbeat_back_addr":"172.21.15.40:6811/117239","heartbeat_front_addr":"172.21.15.40:6810/117239","state":["exists","up"]},{"osd":3,"uuid":"d18ec2f5-7e8f-4732-a523-adc495115cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":117236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":117236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":117236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":117236}]},"public_addr":"172.21.15.40:6812/117236","cluster_addr":"172.21.15.40:6813/117236","heartbeat_back_addr":"172.21.15.40:6815/117236","heartbeat_front_addr":"172.21.15.40:6814/117236","state":["exists","up"]},{"osd":4,"uuid":"4e2465fe-bb27-47bd-8ddf-f334cac6041a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6805","nonce":117131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":117131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":117131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6807","nonce":117131}]},"public_addr":"172.21.15.135:6805/117131","cluster_addr":"172.21.15.135:6806/117131","heartbeat_back_addr":"172.21.15.135:6808/117131","heartbeat_front_addr":"172.21.15.135:6807/117131","state":["exists","up"]},{"osd":5,"uuid":"7ea449db-9b22-48b8-a61b-1937eeeb6c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6801","nonce":117130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":117130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":117130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6803","nonce":117130}]},"public_addr":"172.21.15.135:6801/117130","cluster_addr":"172.21.15.135:6802/117130","heartbeat_back_addr":"172.21.15.135:6804/117130","heartbeat_front_addr":"172.21.15.135:6803/117130","state":["exists","up"]},{"osd":6,"uuid":"c79b4829-6393-4f9a-a9f8-d2aacb505ae9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6813","nonce":117133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":117133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":117133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6815","nonce":117133}]},"public_addr":"172.21.15.135:6813/117133","cluster_addr":"172.21.15.135:6814/117133","heartbeat_back_addr":"172.21.15.135:6816/117133","heartbeat_front_addr":"172.21.15.135:6815/117133","state":["exists","up"]},{"osd":7,"uuid":"4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6809","nonce":117132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":117132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":117132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6811","nonce":117132}]},"public_addr":"172.21.15.135:6809/117132","cluster_addr":"172.21.15.135:6810/117132","heartbeat_back_addr":"172.21.15.135:6812/117132","heartbeat_front_addr":"172.21.15.135:6811/117132","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.725062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.291589+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.713534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.731764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:35.977051+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:36.680016+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:35.693147+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:36.368009+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:47:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.411+0000 7fcbb7aa5700 10 librados: watch_flush enter 2023-09-30T00:47:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.411+0000 7fcbb7aa5700 10 librados: watch_flush exit 2023-09-30T00:47:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.411+0000 7fcbb7aa5700 20 client.4291.objecter shutdown clearing up homeless session... 2023-09-30T00:47:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.411+0000 7fcbb7aa5700 10 client.4291.objecter successfully canceled tick 2023-09-30T00:47:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/3961367194 >> v2:172.21.15.135:6800/116426 conn(0x7fcb9c068890 msgr2=0x7fcb9c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 1 --2- 172.21.15.40:0/3961367194 >> v2:172.21.15.135:6800/116426 conn(0x7fcb9c068890 0x7fcb9c06ad40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcba40091e0 tx=0x7fcba4016000 comp rx=0 tx=0).stop 2023-09-30T00:47:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 10 monclient: shutdown 2023-09-30T00:47:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/3961367194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbb00b1460 msgr2=0x7fcbb00aa440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:55.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 1 --2- 172.21.15.40:0/3961367194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbb00b1460 0x7fcbb00aa440 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcba000cd30 tx=0x7fcba0001b10 comp rx=0 tx=0).stop 2023-09-30T00:47:55.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/3961367194 shutdown_connections 2023-09-30T00:47:55.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 1 --2- 172.21.15.40:0/3961367194 >> v2:172.21.15.135:6800/116426 conn(0x7fcb9c068890 0x7fcb9c06ad40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 1 --2- 172.21.15.40:0/3961367194 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcbb00b1460 0x7fcbb00aa440 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 1 --2- 172.21.15.40:0/3961367194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbb013ac20 0x7fcbb0150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 1 --2- 172.21.15.40:0/3961367194 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcbb00b1830 0x7fcbb013e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/3961367194 >> 172.21.15.40:0/3961367194 conn(0x7fcbb00631f0 msgr2=0x7fcbb00b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:55.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/3961367194 shutdown_connections 2023-09-30T00:47:55.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 1 -- 172.21.15.40:0/3961367194 wait complete. 2023-09-30T00:47:55.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.412+0000 7fcbb7aa5700 1 librados: shutdown 2023-09-30T00:47:55.433 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-30T00:47:55.433 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:47:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.572+0000 7ff8c6f00700 10 monclient: get_monmap_and_config 2023-09-30T00:47:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.572+0000 7ff8c6f00700 10 monclient: build_initial_monmap 2023-09-30T00:47:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.572+0000 7ff8c6f00700 10 monclient: monmap: 2023-09-30T00:47:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:55.573237+0000 2023-09-30T00:47:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:55.573237+0000 2023-09-30T00:47:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:55.573 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:55.574 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:55.574 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:55.574 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:55.574 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:55.574 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:55.574 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:55.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 1 Processor -- start 2023-09-30T00:47:55.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 1 -- start start 2023-09-30T00:47:55.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 10 monclient: init 2023-09-30T00:47:55.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:55.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-09-30T00:47:55.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8c00b1460 0x7ff8c00b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:55.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 10 monclient(hunting): picked mon.noname-b con 0x7ff8c00b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8c00a5fb0 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8c013ac20 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c00a5b20 con 0x7ff8c00a5fb0 2023-09-30T00:47:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c00a71f0 con 0x7ff8c013ac20 2023-09-30T00:47:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00a74d0 con 0x7ff8c00b1460 2023-09-30T00:47:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.573+0000 7ff8c6f00700 10 monclient(hunting): authenticate will time out at 5336.874789s 2023-09-30T00:47:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.574+0000 7ff8c549d700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7ff8c013ac20 0x7ff8c013b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:53968/0 (socket says 172.21.15.40:53968) 2023-09-30T00:47:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.574+0000 7ff8c549d700 1 -- 172.21.15.40:0/1100025726 learned_addr learned my addr 172.21.15.40:0/1100025726 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:55.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.574+0000 7ff8c4c9c700 1 --2- 172.21.15.40:0/1100025726 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8c00b1460 0x7ff8c00b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:55.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.575+0000 7ff8c4c9c700 10 monclient(hunting): get_auth_request con 0x7ff8c00b1460 auth_method 0 2023-09-30T00:47:55.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.575+0000 7ff8c4c9c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:55.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.575+0000 7ff8c4c9c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:55.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.575+0000 7ff8c4c9c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:55.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.575+0000 7ff8c4c9c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:55.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.575+0000 7ff8c4c9c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:55.578 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.575+0000 7ff8c4c9c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:55.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.577+0000 7ff8bf7fe700 1 -- 172.21.15.40:0/1100025726 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51144901 0 0) 0x7ff8c00a71f0 con 0x7ff8c013ac20 2023-09-30T00:47:55.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.577+0000 7ff8bf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:55.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.577+0000 7ff8bf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:55.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.577+0000 7ff8bf7fe700 10 monclient(hunting): my global_id is 4294 2023-09-30T00:47:55.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.577+0000 7ff8bf7fe700 1 -- 172.21.15.40:0/1100025726 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a8003570 con 0x7ff8c013ac20 2023-09-30T00:47:55.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.578+0000 7ff8c4c9c700 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2023-09-30T00:47:55.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.578+0000 7ff8c4c9c700 1 -- 172.21.15.40:0/1100025726 >> v1:172.21.15.135:6789/0 conn(0x7ff8c013ac20 legacy=0x7ff8c013b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:55.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.578+0000 7ff8c4c9c700 1 -- 172.21.15.40:0/1100025726 >> v1:172.21.15.40:6789/0 conn(0x7ff8c00a5fb0 legacy=0x7ff8c0141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.578+0000 7ff8c4c9c700 10 monclient: _finish_hunting 0 2023-09-30T00:47:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.578+0000 7ff8c4c9c700 1 monclient: found mon.noname-b 2023-09-30T00:47:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.578+0000 7ff8c4c9c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.578+0000 7ff8c4c9c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:47:55.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.578+0000 7ff8c4c9c700 1 -- 172.21.15.40:0/1100025726 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c00a63c0 con 0x7ff8c00b1460 2023-09-30T00:47:55.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.580+0000 7ff8c4c9c700 1 --2- 172.21.15.40:0/1100025726 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8c00b1460 0x7ff8c00b1830 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8009e90 tx=0x7ff8b8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f3f325cdf0a8796 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:55.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8bf7fe700 1 -- 172.21.15.40:0/1100025726 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b800e070 con 0x7ff8c00b1460 2023-09-30T00:47:55.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8bf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:55.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8bf7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:55.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8bf7fe700 10 monclient: dump: 2023-09-30T00:47:55.587 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:55.587 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:55.587 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:55.587 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:55.587 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:55.587 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:55.587 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:55.587 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:55.587 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:55.587 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:55.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8bf7fe700 10 monclient: _finish_auth 0 2023-09-30T00:47:55.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8bf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:55.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8bf7fe700 1 -- 172.21.15.40:0/1100025726 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b800bc30 con 0x7ff8c00b1460 2023-09-30T00:47:55.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8bf7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:55.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8bf7fe700 1 -- 172.21.15.40:0/1100025726 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8012430 con 0x7ff8c00b1460 2023-09-30T00:47:55.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8bf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:55.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8bf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:55.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8bf7fe700 10 monclient: dump: 2023-09-30T00:47:55.588 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:55.588 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:55.588 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:55.588 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:55.588 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:55.588 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:55.588 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:55.589 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:55.589 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:55.589 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:55.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8c6f00700 5 monclient: authenticate success, global_id 4232 2023-09-30T00:47:55.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8c6f00700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:55.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.585+0000 7ff8c6f00700 4 set_mon_vals no callback set 2023-09-30T00:47:55.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.586+0000 7ff8c6f00700 10 monclient: shutdown 2023-09-30T00:47:55.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.586+0000 7ff8c6f00700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:55.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.586+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1100025726 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8c00b1460 msgr2=0x7ff8c00b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:55.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.586+0000 7ff8c6f00700 1 --2- 172.21.15.40:0/1100025726 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8c00b1460 0x7ff8c00b1830 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8009e90 tx=0x7ff8b8009780 comp rx=0 tx=0).stop 2023-09-30T00:47:55.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.586+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1100025726 shutdown_connections 2023-09-30T00:47:55.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.586+0000 7ff8c6f00700 1 --2- 172.21.15.40:0/1100025726 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8c00b1460 0x7ff8c00b1830 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.593 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.586+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1100025726 >> 172.21.15.40:0/1100025726 conn(0x7ff8c00631f0 msgr2=0x7ff8c00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:55.593 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.586+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1100025726 shutdown_connections 2023-09-30T00:47:55.593 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.586+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1100025726 wait complete. 2023-09-30T00:47:55.593 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.587+0000 7ff8c6f00700 10 monclient: build_initial_monmap 2023-09-30T00:47:55.593 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.587+0000 7ff8c6f00700 10 monclient: monmap: 2023-09-30T00:47:55.593 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:55.593 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:55.593 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:55.593 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:55.593 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:55.593 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:55.593 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:55.594 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:55.594 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:55.594 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:55.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.587+0000 7ff8c6f00700 1 librados: starting msgr at 2023-09-30T00:47:55.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.587+0000 7ff8c6f00700 1 librados: starting objecter 2023-09-30T00:47:55.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.587+0000 7ff8c6f00700 1 Processor -- start 2023-09-30T00:47:55.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 1 -- start start 2023-09-30T00:47:55.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 1 librados: setting wanted keys 2023-09-30T00:47:55.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 1 librados: calling monclient init 2023-09-30T00:47:55.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 10 monclient: init 2023-09-30T00:47:55.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:55.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:47:55.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8c00a5fb0 0x7ff8c0151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:55.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 10 monclient(hunting): picked mon.noname-a con 0x7ff8c00a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:55.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8c00b1460 0x7ff8c013e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:55.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 10 monclient(hunting): picked mon.noname-b con 0x7ff8c00b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:55.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8c013ac20 0x7ff8c00a9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:55.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 10 monclient(hunting): picked mon.noname-c con 0x7ff8c013ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:55.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:55.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00abee0 con 0x7ff8c00a5fb0 2023-09-30T00:47:55.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:55.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00a9d50 con 0x7ff8c00b1460 2023-09-30T00:47:55.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:55.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c00a8bf0 con 0x7ff8c013ac20 2023-09-30T00:47:55.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:55.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.588+0000 7ff8c6f00700 10 monclient(hunting): authenticate will time out at 5336.889756s 2023-09-30T00:47:55.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.589+0000 7ff8bffff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8c00b1460 0x7ff8c013e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:55.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.589+0000 7ff8c549d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8c013ac20 0x7ff8c00a9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:55.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.589+0000 7ff8c4c9c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8c00a5fb0 0x7ff8c0151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:55.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.589+0000 7ff8bffff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8c00b1460 0x7ff8c013e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43100/0 (socket says 172.21.15.40:43100) 2023-09-30T00:47:55.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.589+0000 7ff8bffff700 1 -- 172.21.15.40:0/1729576462 learned_addr learned my addr 172.21.15.40:0/1729576462 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:55.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.589+0000 7ff8bffff700 10 monclient(hunting): get_auth_request con 0x7ff8c00b1460 auth_method 0 2023-09-30T00:47:55.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.589+0000 7ff8bffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:55.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.589+0000 7ff8bffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:55.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.589+0000 7ff8bffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:55.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.589+0000 7ff8c549d700 10 monclient(hunting): get_auth_request con 0x7ff8c013ac20 auth_method 0 2023-09-30T00:47:55.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.589+0000 7ff8c549d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:55.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.589+0000 7ff8c549d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:55.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.589+0000 7ff8c549d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:55.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.590+0000 7ff8c4c9c700 10 monclient(hunting): get_auth_request con 0x7ff8c00a5fb0 auth_method 0 2023-09-30T00:47:55.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.590+0000 7ff8c4c9c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:55.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.590+0000 7ff8c4c9c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:55.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.590+0000 7ff8c4c9c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:55.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.590+0000 7ff8bffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:55.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.590+0000 7ff8bffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:55.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.590+0000 7ff8bffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:55.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.590+0000 7ff8c549d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:55.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.590+0000 7ff8c549d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:55.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.590+0000 7ff8c549d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:55.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.591+0000 7ff8c4c9c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:55.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.591+0000 7ff8c4c9c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:55.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.591+0000 7ff8c4c9c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:55.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.593+0000 7ff8bffff700 10 monclient(hunting): handle_auth_done global_id 4297 payload 824 2023-09-30T00:47:55.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.593+0000 7ff8bffff700 1 -- 172.21.15.40:0/1729576462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8c013ac20 msgr2=0x7ff8c00a9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:55.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.593+0000 7ff8bffff700 1 --2- 172.21.15.40:0/1729576462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8c013ac20 0x7ff8c00a9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.593+0000 7ff8bffff700 1 -- 172.21.15.40:0/1729576462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8c00a5fb0 msgr2=0x7ff8c0151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:55.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.593+0000 7ff8bffff700 1 --2- 172.21.15.40:0/1729576462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8c00a5fb0 0x7ff8c0151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.593+0000 7ff8bffff700 10 monclient: _finish_hunting 0 2023-09-30T00:47:55.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.593+0000 7ff8bffff700 1 monclient: found mon.noname-b 2023-09-30T00:47:55.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.593+0000 7ff8bffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:55.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.593+0000 7ff8bffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:47:55.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.593+0000 7ff8bffff700 1 -- 172.21.15.40:0/1729576462 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b8009a20 con 0x7ff8c00b1460 2023-09-30T00:47:55.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.596+0000 7ff8bffff700 1 --2- 172.21.15.40:0/1729576462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8c00b1460 0x7ff8c013e880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff8b400c590 tx=0x7ff8b400c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:55.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.596+0000 7ff8bcff9700 10 client.?.objecter ms_handle_connect 0x7ff8c00b1460 2023-09-30T00:47:55.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.596+0000 7ff8bcff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:55.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8bcff9700 1 -- 172.21.15.40:0/1729576462 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4011590 con 0x7ff8c00b1460 2023-09-30T00:47:55.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8bcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:55.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8bcff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:55.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8bcff9700 10 monclient: dump: 2023-09-30T00:47:55.602 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:55.602 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:55.602 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:55.602 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:55.602 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:55.602 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:55.603 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:55.603 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:55.603 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:55.603 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:55.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8bcff9700 10 monclient: _finish_auth 0 2023-09-30T00:47:55.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8bcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:55.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8bcff9700 1 -- 172.21.15.40:0/1729576462 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b4011710 con 0x7ff8c00b1460 2023-09-30T00:47:55.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8c6f00700 5 monclient: authenticate success, global_id 4297 2023-09-30T00:47:55.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8c6f00700 10 monclient: _renew_subs 2023-09-30T00:47:55.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8c6f00700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:55.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1729576462 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c00aa540 con 0x7ff8c00b1460 2023-09-30T00:47:55.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8bcff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:55.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8bcff9700 1 -- 172.21.15.40:0/1729576462 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4011ad0 con 0x7ff8c00b1460 2023-09-30T00:47:55.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8bcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:55.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8bcff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:55.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8bcff9700 10 monclient: dump: 2023-09-30T00:47:55.604 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:55.604 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:55.604 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:55.604 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:55.604 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:55.604 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:55.604 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:55.605 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:55.605 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:55.605 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:55.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8c6f00700 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:55.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8c6f00700 10 monclient: _renew_subs 2023-09-30T00:47:55.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8c6f00700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:55.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1729576462 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c00aaa80 con 0x7ff8c00b1460 2023-09-30T00:47:55.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8c6f00700 1 librados: init done 2023-09-30T00:47:55.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.600+0000 7ff8beffd700 4 set_mon_vals no callback set 2023-09-30T00:47:55.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.601+0000 7ff8c6f00700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:55.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.601+0000 7ff8c6f00700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:55.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.601+0000 7ff8c6f00700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:55.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.601+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1729576462 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c00aa260 con 0x7ff8c00b1460 2023-09-30T00:47:55.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.856+0000 7ff8bcff9700 1 -- 172.21.15.40:0/1729576462 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ff8b401b3b0 con 0x7ff8c00b1460 2023-09-30T00:47:55.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.857+0000 7ff8bcff9700 1 --2- 172.21.15.40:0/1729576462 >> v2:172.21.15.135:6800/116426 conn(0x7ff8ac068910 0x7ff8ac06adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:55.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.857+0000 7ff8bcff9700 1 -- 172.21.15.40:0/1729576462 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff8b4083620 con 0x7ff8c00b1460 2023-09-30T00:47:55.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.857+0000 7ff8bcff9700 10 client.4297.objecter ms_dispatch 0x7ff8c0001550 osd_map(24..24 src has 1..24) v4 2023-09-30T00:47:55.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.857+0000 7ff8bcff9700 3 client.4297.objecter handle_osd_map got epochs [24,24] > 0 2023-09-30T00:47:55.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.857+0000 7ff8bcff9700 3 client.4297.objecter handle_osd_map decoding full epoch 24 2023-09-30T00:47:55.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.857+0000 7ff8bcff9700 20 client.4297.objecter dump_active .. 0 homeless 2023-09-30T00:47:55.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.857+0000 7ff8c4c9c700 1 --2- 172.21.15.40:0/1729576462 >> v2:172.21.15.135:6800/116426 conn(0x7ff8ac068910 0x7ff8ac06adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:55.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.857+0000 7ff8c4c9c700 10 monclient: get_auth_request con 0x7ff8ac068910 auth_method 0 2023-09-30T00:47:55.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.857+0000 7ff8c4c9c700 1 --2- 172.21.15.40:0/1729576462 >> v2:172.21.15.135:6800/116426 conn(0x7ff8ac068910 0x7ff8ac06adc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8009e90 tx=0x7ff8b8019000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:55.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.857+0000 7ff8bcff9700 10 client.4297.objecter ms_handle_connect 0x7ff8ac068910 2023-09-30T00:47:55.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.866+0000 7ff8bcff9700 1 -- 172.21.15.40:0/1729576462 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8b4050ec0 con 0x7ff8c00b1460 2023-09-30T00:47:55.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.866+0000 7ff8bcff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:55.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:55.866+0000 7ff8bcff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:56.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.080+0000 7ff8c6f00700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:56.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.080+0000 7ff8c6f00700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:56.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.080+0000 7ff8c6f00700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:56.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.080+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1729576462 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8c00a9e90 con 0x7ff8c00b1460 2023-09-30T00:47:56.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.407+0000 7ff8bcff9700 1 -- 172.21.15.40:0/1729576462 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+11935 (secure 0 0 0) 0x7ff8b4017020 con 0x7ff8c00b1460 2023-09-30T00:47:56.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.407+0000 7ff8bcff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:47:56.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.407+0000 7ff8bcff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:56.409 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:47:56.409 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"fsid":"238dba7c-9190-48bd-bb67-322ebc9ecbd0","created":"2023-09-30T00:46:45.822318+0000","modified":"2023-09-30T00:47:55.738588+0000","last_up_change":"2023-09-30T00:47:45.793188+0000","last_in_change":"2023-09-30T00:46:52.657791+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:47:40.523068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:47:51.796955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"66e1da65-310e-4463-8aa0-431896832e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":117238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":117238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":117238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":117238}]},"public_addr":"172.21.15.40:6804/117238","cluster_addr":"172.21.15.40:6805/117238","heartbeat_back_addr":"172.21.15.40:6807/117238","heartbeat_front_addr":"172.21.15.40:6806/117238","state":["exists","up"]},{"osd":1,"uuid":"d6dd269f-2eda-4ef7-9da1-2ed5b16291b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":117237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6801","nonce":117237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":117237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":117237}]},"public_addr":"172.21.15.40:6800/117237","cluster_addr":"172.21.15.40:6801/117237","heartbeat_back_addr":"172.21.15.40:6803/117237","heartbeat_front_addr":"172.21.15.40:6802/117237","state":["exists","up"]},{"osd":2,"uuid":"8f0de27d-3674-4ff4-8451-4fcc90786c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":117239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":117239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":117239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":117239}]},"public_addr":"172.21.15.40:6808/117239","cluster_addr":"172.21.15.40:6809/117239","heartbeat_back_addr":"172.21.15.40:6811/117239","heartbeat_front_addr":"172.21.15.40:6810/117239","state":["exists","up"]},{"osd":3,"uuid":"d18ec2f5-7e8f-4732-a523-adc495115cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":117236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":117236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":117236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":117236}]},"public_addr":"172.21.15.40:6812/117236","cluster_addr":"172.21.15.40:6813/117236","heartbeat_back_addr":"172.21.15.40:6815/117236","heartbeat_front_addr":"172.21.15.40:6814/117236","state":["exists","up"]},{"osd":4,"uuid":"4e2465fe-bb27-47bd-8ddf-f334cac6041a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6805","nonce":117131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":117131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":117131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6807","nonce":117131}]},"public_addr":"172.21.15.135:6805/117131","cluster_addr":"172.21.15.135:6806/117131","heartbeat_back_addr":"172.21.15.135:6808/117131","heartbeat_front_addr":"172.21.15.135:6807/117131","state":["exists","up"]},{"osd":5,"uuid":"7ea449db-9b22-48b8-a61b-1937eeeb6c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6801","nonce":117130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":117130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":117130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6803","nonce":117130}]},"public_addr":"172.21.15.135:6801/117130","cluster_addr":"172.21.15.135:6802/117130","heartbeat_back_addr":"172.21.15.135:6804/117130","heartbeat_front_addr":"172.21.15.135:6803/117130","state":["exists","up"]},{"osd":6,"uuid":"c79b4829-6393-4f9a-a9f8-d2aacb505ae9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6813","nonce":117133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":117133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":117133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6815","nonce":117133}]},"public_addr":"172.21.15.135:6813/117133","cluster_addr":"172.21.15.135:6814/117133","heartbeat_back_addr":"172.21.15.135:6816/117133","heartbeat_front_addr":"172.21.15.135:6815/117133","state":["exists","up"]},{"osd":7,"uuid":"4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6809","nonce":117132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":117132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":117132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6811","nonce":117132}]},"public_addr":"172.21.15.135:6809/117132","cluster_addr":"172.21.15.135:6810/117132","heartbeat_back_addr":"172.21.15.135:6812/117132","heartbeat_front_addr":"172.21.15.135:6811/117132","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.725062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.291589+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.713534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.731764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:35.977051+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:36.680016+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:35.693147+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:36.368009+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:47:56.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 10 librados: watch_flush enter 2023-09-30T00:47:56.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 10 librados: watch_flush exit 2023-09-30T00:47:56.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 20 client.4297.objecter shutdown clearing up homeless session... 2023-09-30T00:47:56.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 10 client.4297.objecter successfully canceled tick 2023-09-30T00:47:56.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1729576462 >> v2:172.21.15.135:6800/116426 conn(0x7ff8ac068910 msgr2=0x7ff8ac06adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 1 --2- 172.21.15.40:0/1729576462 >> v2:172.21.15.135:6800/116426 conn(0x7ff8ac068910 0x7ff8ac06adc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8009e90 tx=0x7ff8b8019000 comp rx=0 tx=0).stop 2023-09-30T00:47:56.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 10 monclient: shutdown 2023-09-30T00:47:56.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:56.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1729576462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8c00b1460 msgr2=0x7ff8c013e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 1 --2- 172.21.15.40:0/1729576462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8c00b1460 0x7ff8c013e880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff8b400c590 tx=0x7ff8b400c8a0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1729576462 shutdown_connections 2023-09-30T00:47:56.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 1 --2- 172.21.15.40:0/1729576462 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8c013ac20 0x7ff8c00a9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 1 --2- 172.21.15.40:0/1729576462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8c00a5fb0 0x7ff8c0151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 1 --2- 172.21.15.40:0/1729576462 >> v2:172.21.15.135:6800/116426 conn(0x7ff8ac068910 0x7ff8ac06adc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 1 --2- 172.21.15.40:0/1729576462 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8c00b1460 0x7ff8c013e880 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1729576462 >> 172.21.15.40:0/1729576462 conn(0x7ff8c00631f0 msgr2=0x7ff8c00b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:56.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1729576462 shutdown_connections 2023-09-30T00:47:56.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 1 -- 172.21.15.40:0/1729576462 wait complete. 2023-09-30T00:47:56.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.410+0000 7ff8c6f00700 1 librados: shutdown 2023-09-30T00:47:56.427 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-30T00:47:56.428 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-30T00:47:56.428 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-30T00:47:56.428 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-30T00:47:56.429 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-09-30T00:47:56.429 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-09-30T00:47:56.429 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-09-30T00:47:56.430 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-09-30T00:47:56.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.701+0000 7fa6b00b6700 10 monclient: get_monmap_and_config 2023-09-30T00:47:56.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.701+0000 7fa6b00b6700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.701+0000 7fa6b00b6700 10 monclient: monmap: 2023-09-30T00:47:56.704 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.704 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.704 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:56.702898+0000 2023-09-30T00:47:56.704 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:56.702898+0000 2023-09-30T00:47:56.704 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.704 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.704 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:56.704 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:56.705 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:56.705 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:56.705 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:56.705 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.703+0000 7f59eede8700 10 monclient: get_monmap_and_config 2023-09-30T00:47:56.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.703+0000 7f59eede8700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.703+0000 7f59eede8700 10 monclient: monmap: 2023-09-30T00:47:56.706 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.706 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.706 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:56.704338+0000 2023-09-30T00:47:56.706 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:56.704338+0000 2023-09-30T00:47:56.706 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.706 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.706 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:56.706 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:56.706 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:56.707 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:56.707 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:56.707 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.704+0000 7f4cfcfba700 10 monclient: get_monmap_and_config 2023-09-30T00:47:56.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.704+0000 7f4cfcfba700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.704+0000 7f4cfcfba700 10 monclient: monmap: 2023-09-30T00:47:56.708 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.708 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.708 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:56.705602+0000 2023-09-30T00:47:56.708 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:56.705602+0000 2023-09-30T00:47:56.708 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.708 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.708 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:56.708 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:56.708 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:56.708 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:56.708 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:56.708 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.705+0000 7f21e5466700 10 monclient: get_monmap_and_config 2023-09-30T00:47:56.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.705+0000 7f21e5466700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.705+0000 7f21e5466700 10 monclient: monmap: 2023-09-30T00:47:56.709 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.709 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:56.706153+0000 2023-09-30T00:47:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:56.706153+0000 2023-09-30T00:47:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:56.710 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.706+0000 7f4cfcfba700 1 Processor -- start 2023-09-30T00:47:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.706+0000 7f59eede8700 1 Processor -- start 2023-09-30T00:47:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.706+0000 7fa6b00b6700 1 Processor -- start 2023-09-30T00:47:56.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.706+0000 7f4cfcfba700 1 -- start start 2023-09-30T00:47:56.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.706+0000 7f4cfcfba700 10 monclient: init 2023-09-30T00:47:56.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f4cfcfba700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f4cfcfba700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-09-30T00:47:56.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f4cfcfba700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4cf806c580 0x7f4cf806e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f4cfcfba700 10 monclient(hunting): picked mon.noname-a con 0x7f4cf806c580 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:56.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f4cfcfba700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4cf806c950 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:56.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f4cfcfba700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4cf813a4f0 0x7f4cf813a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f4cfcfba700 10 monclient(hunting): picked mon.noname-b con 0x7f4cf813a4f0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f4cfcfba700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf806dc50 con 0x7f4cf806c950 2023-09-30T00:47:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f4cfcfba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f4cfcfba700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf806cf10 con 0x7f4cf806c580 2023-09-30T00:47:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f4cfcfba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f4cfcfba700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf806bc40 con 0x7f4cf813a4f0 2023-09-30T00:47:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f4cfcfba700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f4cfcfba700 10 monclient(hunting): authenticate will time out at 5338.008582s 2023-09-30T00:47:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f59eede8700 1 -- start start 2023-09-30T00:47:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.707+0000 7f59eede8700 10 monclient: init 2023-09-30T00:47:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.708+0000 7f59eede8700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.708+0000 7f59eede8700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-09-30T00:47:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.708+0000 7f59eede8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59e806c580 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.708+0000 7f59eede8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59e806c950 0x7f59e806e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.708+0000 7f59eede8700 10 monclient(hunting): picked mon.noname-b con 0x7f59e806c950 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.708+0000 7f59eede8700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f59e813a4f0 0x7f59e813a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.708+0000 7f59eede8700 10 monclient(hunting): picked mon.noname-c con 0x7f59e813a4f0 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.708+0000 7f59eede8700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e806dc50 con 0x7f59e806c580 2023-09-30T00:47:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.708+0000 7f59eede8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.708+0000 7f59eede8700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e806cf10 con 0x7f59e813a4f0 2023-09-30T00:47:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.708+0000 7f59eede8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.708+0000 7f59eede8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e806bc40 con 0x7f59e806c950 2023-09-30T00:47:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.708+0000 7f59eede8700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.708+0000 7f59eede8700 10 monclient(hunting): authenticate will time out at 5338.009907s 2023-09-30T00:47:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.709+0000 7f4cf5d9b700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4cf806c950 0x7f4cf806e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43282/0 (socket says 172.21.15.40:43282) 2023-09-30T00:47:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.709+0000 7f4cf5d9b700 1 -- 172.21.15.40:0/2080520796 learned_addr learned my addr 172.21.15.40:0/2080520796 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7f4cf659c700 1 --2- 172.21.15.40:0/2080520796 >> v2:172.21.15.40:3300/0 conn(0x7f4cf806c580 0x7f4cf806e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7f4cf659c700 10 monclient(hunting): get_auth_request con 0x7f4cf806c580 auth_method 0 2023-09-30T00:47:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7f4cf659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7f4cf659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7f4cf659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7f21e5466700 1 Processor -- start 2023-09-30T00:47:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.709+0000 7fa6b00b6700 1 -- start start 2023-09-30T00:47:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.709+0000 7fa6b00b6700 10 monclient: init 2023-09-30T00:47:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-09-30T00:47:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa6a806c580 0x7fa6a806e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 10 monclient(hunting): picked mon.noname-a con 0x7fa6a806c580 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6a806c950 0x7fa6a806e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 10 monclient(hunting): picked mon.noname-b con 0x7fa6a806c950 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa6a813d8a0 0x7fa6a813dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 10 monclient(hunting): picked mon.noname-c con 0x7fa6a813d8a0 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a806cf10 con 0x7fa6a806c580 2023-09-30T00:47:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a806bc40 con 0x7fa6a813d8a0 2023-09-30T00:47:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a806dc50 con 0x7fa6a806c950 2023-09-30T00:47:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7fa6b00b6700 10 monclient(hunting): authenticate will time out at 5338.011824s 2023-09-30T00:47:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f4cf659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f4cf659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f4cf659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7f21e5466700 1 -- start start 2023-09-30T00:47:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7f21e5466700 10 monclient: init 2023-09-30T00:47:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f21e5466700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f21e5466700 10 monclient: _add_conns ranks=[2,1,4,3,0] 2023-09-30T00:47:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f21e5466700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21e006c580 0x7f21e006e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f21e5466700 10 monclient(hunting): picked mon.noname-b con 0x7f21e006c580 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f21e5466700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f21e006c950 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f21e5466700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f21e013a4f0 0x7f21e013a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f21e5466700 10 monclient(hunting): picked mon.noname-c con 0x7f21e013a4f0 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f21e5466700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e006dc50 con 0x7f21e006c950 2023-09-30T00:47:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f21e5466700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f21e5466700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e006cf10 con 0x7f21e013a4f0 2023-09-30T00:47:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f21e5466700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f21e5466700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e006bc40 con 0x7f21e006c580 2023-09-30T00:47:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f21e5466700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7f21e5466700 10 monclient(hunting): authenticate will time out at 5338.012847s 2023-09-30T00:47:56.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.711+0000 7fa6ae653700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa6a813d8a0 0x7fa6a813dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7f59ecb84700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f59e806c580 0x7f59e806e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:53972/0 (socket says 172.21.15.40:53972) 2023-09-30T00:47:56.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.710+0000 7f59ecb84700 1 -- 172.21.15.40:0/3608755992 learned_addr learned my addr 172.21.15.40:0/3608755992 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:56.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.712+0000 7f59e7fff700 1 --2- 172.21.15.40:0/3608755992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59e806c950 0x7f59e806e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.712+0000 7f59e7fff700 10 monclient(hunting): get_auth_request con 0x7f59e806c950 auth_method 0 2023-09-30T00:47:56.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.712+0000 7f59e7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.712+0000 7f59e7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.712+0000 7f59e7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.714+0000 7fa6ae653700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa6a813d8a0 0x7fa6a813dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43118/0 (socket says 172.21.15.40:43118) 2023-09-30T00:47:56.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.714+0000 7fa6ae653700 1 -- 172.21.15.40:0/3203612613 learned_addr learned my addr 172.21.15.40:0/3203612613 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:56.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.714+0000 7fa6ae653700 10 monclient(hunting): get_auth_request con 0x7fa6a813d8a0 auth_method 0 2023-09-30T00:47:56.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.714+0000 7fa6ae653700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.714+0000 7fa6ae653700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.714+0000 7fa6ae653700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.714+0000 7f59ed385700 1 --2- 172.21.15.40:0/3608755992 >> v2:172.21.15.135:3300/0 conn(0x7f59e813a4f0 0x7f59e813a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.714+0000 7f4cf6d9d700 1 --2- 172.21.15.40:0/2080520796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4cf813a4f0 0x7f4cf813a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.714+0000 7f4cf6d9d700 10 monclient(hunting): get_auth_request con 0x7f4cf813a4f0 auth_method 0 2023-09-30T00:47:56.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.714+0000 7f4cf6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.714+0000 7f4cf6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.714+0000 7f4cf6d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.715+0000 7fa6ae653700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.715+0000 7fa6ae653700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.715+0000 7fa6ae653700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.715+0000 7fa6ad651700 1 --2- 172.21.15.40:0/3203612613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6a806c950 0x7fa6a806e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.715+0000 7fa6ad651700 10 monclient(hunting): get_auth_request con 0x7fa6a806c950 auth_method 0 2023-09-30T00:47:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.715+0000 7fa6ad651700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.715+0000 7fa6ad651700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.715+0000 7fa6ad651700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.716+0000 7f4cf659c700 10 monclient(hunting): handle_auth_done global_id 4278 payload 274 2023-09-30T00:47:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.716+0000 7f4cf659c700 1 -- 172.21.15.40:0/2080520796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4cf813a4f0 msgr2=0x7f4cf813a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.716+0000 7f4cf659c700 1 --2- 172.21.15.40:0/2080520796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4cf813a4f0 0x7f4cf813a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.716+0000 7f4cf659c700 1 -- 172.21.15.40:0/2080520796 >> v1:172.21.15.40:6789/0 conn(0x7f4cf806c950 legacy=0x7f4cf806e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.716+0000 7f4cf659c700 10 monclient: _finish_hunting 0 2023-09-30T00:47:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.716+0000 7f4cf659c700 1 monclient: found mon.noname-a 2023-09-30T00:47:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.716+0000 7f4cf659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.716+0000 7f4cf659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.716+0000 7f4cf659c700 1 -- 172.21.15.40:0/2080520796 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf80acc00 con 0x7f4cf806c580 2023-09-30T00:47:56.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.718+0000 7fa6ae653700 10 monclient(hunting): handle_auth_done global_id 4306 payload 274 2023-09-30T00:47:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.718+0000 7fa6ae653700 1 -- 172.21.15.40:0/3203612613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6a806c950 msgr2=0x7fa6a806e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.718+0000 7fa6ae653700 1 --2- 172.21.15.40:0/3203612613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6a806c950 0x7fa6a806e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.718+0000 7fa6ae653700 1 -- 172.21.15.40:0/3203612613 >> v2:172.21.15.40:3300/0 conn(0x7fa6a806c580 msgr2=0x7fa6a806e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.718+0000 7fa6ae653700 1 --2- 172.21.15.40:0/3203612613 >> v2:172.21.15.40:3300/0 conn(0x7fa6a806c580 0x7fa6a806e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.718+0000 7fa6ae653700 10 monclient: _finish_hunting 0 2023-09-30T00:47:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.718+0000 7fa6ae653700 1 monclient: found mon.noname-c 2023-09-30T00:47:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.718+0000 7fa6ae653700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.718+0000 7fa6ae653700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:47:56.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.718+0000 7fa6ae653700 1 -- 172.21.15.40:0/3203612613 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a80acc00 con 0x7fa6a813d8a0 2023-09-30T00:47:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.719+0000 7f59ed385700 10 monclient(hunting): get_auth_request con 0x7f59e813a4f0 auth_method 0 2023-09-30T00:47:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.719+0000 7f59ed385700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.719+0000 7f59ed385700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.719+0000 7f59ed385700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.720+0000 7f59ed385700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.720+0000 7f59ed385700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.720+0000 7f59ed385700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.720+0000 7f59e77fe700 1 -- 172.21.15.40:0/3608755992 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565851752 0 0) 0x7f59e806dc50 con 0x7f59e806c580 2023-09-30T00:47:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.720+0000 7f59e77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.720+0000 7f59e77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.720+0000 7f59e77fe700 10 monclient(hunting): my global_id is 4303 2023-09-30T00:47:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.720+0000 7f59e77fe700 1 -- 172.21.15.40:0/3608755992 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d0003570 con 0x7f59e806c580 2023-09-30T00:47:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.721+0000 7f63f3635700 10 monclient: get_monmap_and_config 2023-09-30T00:47:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.721+0000 7f63f3635700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.721+0000 7f63f3635700 10 monclient: monmap: 2023-09-30T00:47:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:56.722216+0000 2023-09-30T00:47:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:56.722216+0000 2023-09-30T00:47:56.727 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:56.728 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.721+0000 7f4cf659c700 1 --2- 172.21.15.40:0/2080520796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cf806c580 0x7f4cf806e2e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009910 tx=0x7f4cec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed52cc6ff8927cd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.722+0000 7f59e7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.722+0000 7f59e7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.722+0000 7f59e7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.722+0000 7f63f3635700 1 Processor -- start 2023-09-30T00:47:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.723+0000 7f63f3635700 1 -- start start 2023-09-30T00:47:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.723+0000 7f63f3635700 10 monclient: init 2023-09-30T00:47:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7f63f3635700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7f63f3635700 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-09-30T00:47:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7f63f3635700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63ec0a5710 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7f63f3635700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f63ec13e1d0 0x7f63ec13e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7f63f3635700 10 monclient(hunting): picked mon.noname-a con 0x7f63ec13e1d0 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7f63f3635700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63ec13eb60 0x7f63ec13ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7f63f3635700 10 monclient(hunting): picked mon.noname-b con 0x7f63ec13eb60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7f63f3635700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63ec0a5ae0 con 0x7f63ec0a5710 2023-09-30T00:47:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7f63f3635700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7f63f3635700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63ec0a7ff0 con 0x7f63ec13e1d0 2023-09-30T00:47:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7f63f3635700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7f63f3635700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec0a7cf0 con 0x7f63ec13eb60 2023-09-30T00:47:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7f63f3635700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7f63f3635700 10 monclient(hunting): authenticate will time out at 5338.025229s 2023-09-30T00:47:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.724+0000 7fa6ae653700 1 --2- 172.21.15.40:0/3203612613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a813d8a0 0x7fa6a813dd20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8069c20 tx=0x7fa698009ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc84a13cdcdba132 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.726+0000 7f59e7fff700 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2023-09-30T00:47:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.726+0000 7f59e7fff700 1 -- 172.21.15.40:0/3608755992 >> v2:172.21.15.135:3300/0 conn(0x7f59e813a4f0 msgr2=0x7f59e813a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.726+0000 7f59e7fff700 1 --2- 172.21.15.40:0/3608755992 >> v2:172.21.15.135:3300/0 conn(0x7f59e813a4f0 0x7f59e813a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.726+0000 7f59e7fff700 1 -- 172.21.15.40:0/3608755992 >> v1:172.21.15.135:6789/0 conn(0x7f59e806c580 legacy=0x7f59e806e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.726+0000 7f59e7fff700 10 monclient: _finish_hunting 0 2023-09-30T00:47:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.726+0000 7f59e7fff700 1 monclient: found mon.noname-b 2023-09-30T00:47:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.726+0000 7f59e7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.726+0000 7f59e7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:47:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.726+0000 7f59e7fff700 1 -- 172.21.15.40:0/3608755992 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e80acc00 con 0x7f59e806c950 2023-09-30T00:47:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.725+0000 7f21df7fe700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f21e013a4f0 0x7f21e013a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.727+0000 7f21df7fe700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f21e013a4f0 0x7f21e013a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43126/0 (socket says 172.21.15.40:43126) 2023-09-30T00:47:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.727+0000 7f21df7fe700 1 -- 172.21.15.40:0/25882841 learned_addr learned my addr 172.21.15.40:0/25882841 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.727+0000 7f21df7fe700 10 monclient(hunting): get_auth_request con 0x7f21e013a4f0 auth_method 0 2023-09-30T00:47:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.727+0000 7f21df7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.727+0000 7f21df7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.727+0000 7f21df7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.727+0000 7f9c8bdf5700 10 monclient: get_monmap_and_config 2023-09-30T00:47:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.727+0000 7f9c8bdf5700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.727+0000 7f9c8bdf5700 10 monclient: monmap: 2023-09-30T00:47:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:56.728724+0000 2023-09-30T00:47:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:56.728724+0000 2023-09-30T00:47:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:56.734 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.729+0000 7f9c8bdf5700 1 Processor -- start 2023-09-30T00:47:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.729+0000 7f21df7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.729+0000 7f21df7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.729+0000 7f21df7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.729+0000 7f9c8bdf5700 1 -- start start 2023-09-30T00:47:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.729+0000 7f9c8bdf5700 10 monclient: init 2023-09-30T00:47:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.730+0000 7f9c8bdf5700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.730+0000 7f9c8bdf5700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-09-30T00:47:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.730+0000 7f9c8bdf5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c840b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.730+0000 7f9c8bdf5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c840b1830 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.730+0000 7f9c8bdf5700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9c840a5fb0 0x7f9c8413b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.730+0000 7f9c8bdf5700 10 monclient(hunting): picked mon.noname-c con 0x7f9c840a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.730+0000 7f9c8bdf5700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c840a5b20 con 0x7f9c840b1830 2023-09-30T00:47:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.730+0000 7f9c8bdf5700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c840a71f0 con 0x7f9c840b1460 2023-09-30T00:47:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.730+0000 7f9c8bdf5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.730+0000 7f9c8bdf5700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c84141e40 con 0x7f9c840a5fb0 2023-09-30T00:47:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.730+0000 7f9c8bdf5700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.730+0000 7f9c8bdf5700 10 monclient(hunting): authenticate will time out at 5338.031273s 2023-09-30T00:47:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.730+0000 7f21deffd700 1 --2- 172.21.15.40:0/25882841 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21e006c580 0x7f21e006e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.731+0000 7f21deffd700 10 monclient(hunting): get_auth_request con 0x7f21e006c580 auth_method 0 2023-09-30T00:47:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.731+0000 7f21deffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.731+0000 7f21deffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.731+0000 7f21deffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.731+0000 7f21df7fe700 10 monclient(hunting): handle_auth_done global_id 4312 payload 274 2023-09-30T00:47:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.731+0000 7f21df7fe700 1 -- 172.21.15.40:0/25882841 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21e006c580 msgr2=0x7f21e006e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.731+0000 7f21df7fe700 1 --2- 172.21.15.40:0/25882841 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21e006c580 0x7f21e006e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.731+0000 7f21df7fe700 1 -- 172.21.15.40:0/25882841 >> v1:172.21.15.135:6789/0 conn(0x7f21e006c950 legacy=0x7f21e006e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.731+0000 7f21df7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:47:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.731+0000 7f21df7fe700 1 monclient: found mon.noname-c 2023-09-30T00:47:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.731+0000 7f21df7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.731+0000 7f21df7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:47:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.731+0000 7f21df7fe700 1 -- 172.21.15.40:0/25882841 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e00acc00 con 0x7f21e013a4f0 2023-09-30T00:47:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.733+0000 7f9c89390700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9c840b1830 0x7f9c8413ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43298/0 (socket says 172.21.15.40:43298) 2023-09-30T00:47:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.733+0000 7f9c89390700 1 -- 172.21.15.40:0/1953156082 learned_addr learned my addr 172.21.15.40:0/1953156082 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.734+0000 7f59e7fff700 1 --2- 172.21.15.40:0/3608755992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59e806c950 0x7f59e806e9f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00bb80 tx=0x7f59dc00b420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40211e897c3436fa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.736+0000 7f4cf559a700 1 -- 172.21.15.40:0/2080520796 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec013a90 con 0x7f4cf806c580 2023-09-30T00:47:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.736+0000 7f4cf559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.736+0000 7f4cf559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.736+0000 7f4cf559a700 10 monclient: dump: 2023-09-30T00:47:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.740 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.736+0000 7f4cf559a700 10 monclient: _finish_auth 0 2023-09-30T00:47:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.736+0000 7f4cf559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.736+0000 7f4cf559a700 1 -- 172.21.15.40:0/2080520796 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cec013bf0 con 0x7f4cf806c580 2023-09-30T00:47:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.736+0000 7f4cf559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.736+0000 7f4cf559a700 1 -- 172.21.15.40:0/2080520796 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec00b610 con 0x7f4cf806c580 2023-09-30T00:47:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.736+0000 7f4cf559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.736+0000 7f4cf559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.736+0000 7f4cf559a700 10 monclient: dump: 2023-09-30T00:47:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.742 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.737+0000 7f21df7fe700 1 --2- 172.21.15.40:0/25882841 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e013a4f0 0x7f21e013a970 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009350 tx=0x7f21d0009ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac3b9050441364fb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.737+0000 7f4cfcfba700 5 monclient: authenticate success, global_id 4278 2023-09-30T00:47:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.737+0000 7f4cfcfba700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.737+0000 7f4cfcfba700 4 set_mon_vals no callback set 2023-09-30T00:47:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.738+0000 7f4cfcfba700 10 monclient: shutdown 2023-09-30T00:47:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.738+0000 7f4cfcfba700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.738+0000 7f4cfcfba700 1 -- 172.21.15.40:0/2080520796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cf806c580 msgr2=0x7f4cf806e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.738+0000 7f4cfcfba700 1 --2- 172.21.15.40:0/2080520796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cf806c580 0x7f4cf806e2e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009910 tx=0x7f4cec0091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.738+0000 7f4cfcfba700 1 -- 172.21.15.40:0/2080520796 shutdown_connections 2023-09-30T00:47:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.738+0000 7f4cfcfba700 1 --2- 172.21.15.40:0/2080520796 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4cf813a4f0 0x7f4cf813a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.738+0000 7f4cfcfba700 1 --2- 172.21.15.40:0/2080520796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cf806c580 0x7f4cf806e2e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.738+0000 7f4cfcfba700 1 -- 172.21.15.40:0/2080520796 >> 172.21.15.40:0/2080520796 conn(0x7f4cf8067bf0 msgr2=0x7f4cf80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6ace50700 1 -- 172.21.15.40:0/3203612613 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69800c970 con 0x7fa6a813d8a0 2023-09-30T00:47:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6ace50700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6ace50700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6ace50700 10 monclient: dump: 2023-09-30T00:47:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.745 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6ace50700 10 monclient: _finish_auth 0 2023-09-30T00:47:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6ace50700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6ace50700 1 -- 172.21.15.40:0/3203612613 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69800cad0 con 0x7fa6a813d8a0 2023-09-30T00:47:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6b00b6700 5 monclient: authenticate success, global_id 4306 2023-09-30T00:47:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6b00b6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6ace50700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6b00b6700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6ace50700 1 -- 172.21.15.40:0/3203612613 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69800ce70 con 0x7fa6a813d8a0 2023-09-30T00:47:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6b00b6700 4 set_mon_vals no callback set 2023-09-30T00:47:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6ace50700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6ace50700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.739+0000 7fa6ace50700 10 monclient: dump: 2023-09-30T00:47:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.747 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.747 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.740+0000 7fa6b00b6700 10 monclient: shutdown 2023-09-30T00:47:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.740+0000 7fa6b00b6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.740+0000 7fa6b00b6700 1 -- 172.21.15.40:0/3203612613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a813d8a0 msgr2=0x7fa6a813dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.740+0000 7fa6b00b6700 1 --2- 172.21.15.40:0/3203612613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a813d8a0 0x7fa6a813dd20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8069c20 tx=0x7fa698009ae0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.740+0000 7fa6b00b6700 1 -- 172.21.15.40:0/3203612613 shutdown_connections 2023-09-30T00:47:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.740+0000 7fa6b00b6700 1 --2- 172.21.15.40:0/3203612613 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a813d8a0 0x7fa6a813dd20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.740+0000 7fa6b00b6700 1 --2- 172.21.15.40:0/3203612613 >> v2:172.21.15.40:3300/0 conn(0x7fa6a806c580 0x7fa6a806e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.740+0000 7fa6b00b6700 1 --2- 172.21.15.40:0/3203612613 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6a806c950 0x7fa6a806e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.740+0000 7fa6b00b6700 1 -- 172.21.15.40:0/3203612613 >> 172.21.15.40:0/3203612613 conn(0x7fa6a8067bf0 msgr2=0x7fa6a80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f4cfcfba700 1 -- 172.21.15.40:0/2080520796 shutdown_connections 2023-09-30T00:47:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f9c88b8f700 1 -- 172.21.15.40:0/1953156082 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822821818 0 0) 0x7f9c840a5b20 con 0x7f9c840b1830 2023-09-30T00:47:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f9c88b8f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f9c88b8f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f9c88b8f700 10 monclient(hunting): my global_id is 4281 2023-09-30T00:47:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f9c88b8f700 1 -- 172.21.15.40:0/1953156082 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c70003570 con 0x7f9c840b1830 2023-09-30T00:47:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.742+0000 7f9c8a392700 1 --2- 172.21.15.40:0/1953156082 >> v2:172.21.15.135:3300/0 conn(0x7f9c840a5fb0 0x7f9c8413b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f4cfcfba700 1 -- 172.21.15.40:0/2080520796 wait complete. 2023-09-30T00:47:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.742+0000 7f4cfcfba700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.742+0000 7f4cfcfba700 10 monclient: monmap: 2023-09-30T00:47:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:56.751 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.742+0000 7f4cfcfba700 1 librados: starting msgr at 2023-09-30T00:47:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.742+0000 7f4cfcfba700 1 librados: starting objecter 2023-09-30T00:47:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.742+0000 7f4cfcfba700 1 Processor -- start 2023-09-30T00:47:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.743+0000 7f4cfcfba700 1 -- start start 2023-09-30T00:47:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.743+0000 7f4cfcfba700 1 librados: setting wanted keys 2023-09-30T00:47:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.743+0000 7f4cfcfba700 1 librados: calling monclient init 2023-09-30T00:47:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.743+0000 7f4cfcfba700 10 monclient: init 2023-09-30T00:47:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.743+0000 7f4cfcfba700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.743+0000 7f4cfcfba700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:47:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.743+0000 7f4cfcfba700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4cf806c580 0x7f4cf813d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.743+0000 7f4cfcfba700 10 monclient(hunting): picked mon.noname-c con 0x7f4cf806c580 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.743+0000 7f4cfcfba700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cf806c950 0x7f4cf80a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.743+0000 7f4cfcfba700 10 monclient(hunting): picked mon.noname-b con 0x7f4cf806c950 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.743+0000 7f4cfcfba700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cf813a4f0 0x7f4cf815a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.743+0000 7f4cfcfba700 10 monclient(hunting): picked mon.noname-a con 0x7f4cf813a4f0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.743+0000 7f4cfcfba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.743+0000 7f4cfcfba700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf813db00 con 0x7f4cf813a4f0 2023-09-30T00:47:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.744+0000 7f4cfcfba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.744+0000 7f4cfcfba700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf813e300 con 0x7f4cf806c950 2023-09-30T00:47:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.744+0000 7f4cfcfba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.744+0000 7f4cfcfba700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf80a9b10 con 0x7f4cf806c580 2023-09-30T00:47:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.744+0000 7f4cfcfba700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.744+0000 7f4cfcfba700 10 monclient(hunting): authenticate will time out at 5338.045026s 2023-09-30T00:47:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f63f0bd0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f63ec13e1d0 0x7f63ec13e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f63f1bd2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63ec13eb60 0x7f63ec13ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f63f1bd2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63ec13eb60 0x7f63ec13ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36442/0 (socket says 172.21.15.40:36442) 2023-09-30T00:47:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f63f1bd2700 1 -- 172.21.15.40:0/3030148126 learned_addr learned my addr 172.21.15.40:0/3030148126 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f63f1bd2700 10 monclient(hunting): get_auth_request con 0x7f63ec13eb60 auth_method 0 2023-09-30T00:47:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f63f1bd2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f63f1bd2700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.741+0000 7f63f1bd2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.742+0000 7f63f1bd2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.742+0000 7f63f1bd2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.742+0000 7f63f1bd2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.744+0000 7f63f0bd0700 10 monclient(hunting): get_auth_request con 0x7f63ec13e1d0 auth_method 0 2023-09-30T00:47:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.744+0000 7f63f0bd0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.744+0000 7f63f0bd0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.744+0000 7f63f0bd0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.745+0000 7f63f1bd2700 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2023-09-30T00:47:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.745+0000 7f63f1bd2700 1 -- 172.21.15.40:0/3030148126 >> v2:172.21.15.40:3300/0 conn(0x7f63ec13e1d0 msgr2=0x7f63ec13e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.745+0000 7f63f1bd2700 1 --2- 172.21.15.40:0/3030148126 >> v2:172.21.15.40:3300/0 conn(0x7f63ec13e1d0 0x7f63ec13e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.745+0000 7f63f1bd2700 1 -- 172.21.15.40:0/3030148126 >> v1:172.21.15.40:6789/0 conn(0x7f63ec0a5710 legacy=0x7f63ec0a75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.745+0000 7f63f1bd2700 10 monclient: _finish_hunting 0 2023-09-30T00:47:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.745+0000 7f63f1bd2700 1 monclient: found mon.noname-b 2023-09-30T00:47:56.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.745+0000 7f63f1bd2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:56.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.745+0000 7f63f1bd2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:47:56.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.745+0000 7f63f1bd2700 1 -- 172.21.15.40:0/3030148126 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ec0aedc0 con 0x7f63ec13eb60 2023-09-30T00:47:56.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7fa6b00b6700 1 -- 172.21.15.40:0/3203612613 shutdown_connections 2023-09-30T00:47:56.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7fa6b00b6700 1 -- 172.21.15.40:0/3203612613 wait complete. 2023-09-30T00:47:56.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7f4cf659c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4cf806c580 0x7f4cf813d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7fa6b00b6700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7fa6b00b6700 10 monclient: monmap: 2023-09-30T00:47:56.757 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.757 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:56.757 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:56.757 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:56.757 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.757 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:56.758 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:56.758 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:56.758 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:56.758 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.747+0000 7fa6b00b6700 1 librados: starting msgr at 2023-09-30T00:47:56.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.747+0000 7fa6b00b6700 1 librados: starting objecter 2023-09-30T00:47:56.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.747+0000 7fa6b00b6700 1 Processor -- start 2023-09-30T00:47:56.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.747+0000 7f4cf659c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4cf806c580 0x7f4cf813d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36452/0 (socket says 172.21.15.40:36452) 2023-09-30T00:47:56.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.747+0000 7f4cf659c700 1 -- 172.21.15.40:0/4045507777 learned_addr learned my addr 172.21.15.40:0/4045507777 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:56.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.747+0000 7f4cf659c700 10 monclient(hunting): get_auth_request con 0x7f4cf806c580 auth_method 0 2023-09-30T00:47:56.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.747+0000 7f4cf659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.747+0000 7f4cf659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.747+0000 7f4cf659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.747+0000 7fa6b00b6700 1 -- start start 2023-09-30T00:47:56.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.747+0000 7fa6b00b6700 1 librados: setting wanted keys 2023-09-30T00:47:56.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.747+0000 7fa6b00b6700 1 librados: calling monclient init 2023-09-30T00:47:56.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.747+0000 7fa6b00b6700 10 monclient: init 2023-09-30T00:47:56.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:47:56.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6a806c580 0x7fa6a80a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 10 monclient(hunting): picked mon.noname-c con 0x7fa6a806c580 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:56.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6a806c950 0x7fa6a8118520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 10 monclient(hunting): picked mon.noname-a con 0x7fa6a806c950 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:56.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a813d8a0 0x7fa6a80a7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 10 monclient(hunting): picked mon.noname-b con 0x7fa6a813d8a0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:56.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a8119e70 con 0x7fa6a806c950 2023-09-30T00:47:56.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a80a7a80 con 0x7fa6a813d8a0 2023-09-30T00:47:56.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a8118a90 con 0x7fa6a806c580 2023-09-30T00:47:56.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7fa6b00b6700 10 monclient(hunting): authenticate will time out at 5338.049633s 2023-09-30T00:47:56.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7f4cf659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7f4cf659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7f4cf659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.744+0000 7f9c8a392700 10 monclient(hunting): get_auth_request con 0x7f9c840a5fb0 auth_method 0 2023-09-30T00:47:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.744+0000 7f9c8a392700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.744+0000 7f9c8a392700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.744+0000 7f9c8a392700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7f9c8a392700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7f9c8a392700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.748+0000 7f9c8a392700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.745+0000 7f21ddffb700 1 -- 172.21.15.40:0/25882841 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d000c8f0 con 0x7f21e013a4f0 2023-09-30T00:47:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7f21ddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7f21ddffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:56.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7f21ddffb700 10 monclient: dump: 2023-09-30T00:47:56.763 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.763 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.763 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.763 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.763 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.763 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.763 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.763 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.763 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.763 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7f21ddffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7f21ddffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7f21ddffb700 1 -- 172.21.15.40:0/25882841 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d000ca50 con 0x7f21e013a4f0 2023-09-30T00:47:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7f21ddffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7f21ddffb700 1 -- 172.21.15.40:0/25882841 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d000cdf0 con 0x7f21e013a4f0 2023-09-30T00:47:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7f21ddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7f21ddffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.746+0000 7f21ddffb700 10 monclient: dump: 2023-09-30T00:47:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.764 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.765 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.750+0000 7f63f1bd2700 1 --2- 172.21.15.40:0/3030148126 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63ec13eb60 0x7f63ec13ae40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f63dc0099b0 tx=0x7f63dc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3211556657b6d9c9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.749+0000 7f21e5466700 5 monclient: authenticate success, global_id 4312 2023-09-30T00:47:56.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.749+0000 7f21e5466700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:56.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.749+0000 7f21e5466700 4 set_mon_vals no callback set 2023-09-30T00:47:56.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.750+0000 7f21e5466700 10 monclient: shutdown 2023-09-30T00:47:56.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.750+0000 7f21e5466700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:56.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.750+0000 7f21e5466700 1 -- 172.21.15.40:0/25882841 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e013a4f0 msgr2=0x7f21e013a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.750+0000 7f21e5466700 1 --2- 172.21.15.40:0/25882841 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e013a4f0 0x7f21e013a970 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f21d0009350 tx=0x7f21d0009ae0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.751+0000 7f21e5466700 1 -- 172.21.15.40:0/25882841 shutdown_connections 2023-09-30T00:47:56.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.751+0000 7f21e5466700 1 --2- 172.21.15.40:0/25882841 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21e006c580 0x7f21e006e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.751+0000 7f21e5466700 1 --2- 172.21.15.40:0/25882841 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e013a4f0 0x7f21e013a970 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.751+0000 7f21e5466700 1 -- 172.21.15.40:0/25882841 >> 172.21.15.40:0/25882841 conn(0x7f21e0067bf0 msgr2=0x7f21e00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:56.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.751+0000 7f21e5466700 1 -- 172.21.15.40:0/25882841 shutdown_connections 2023-09-30T00:47:56.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.752+0000 7f9c8a392700 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2023-09-30T00:47:56.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.752+0000 7f9c8a392700 1 -- 172.21.15.40:0/1953156082 >> v1:172.21.15.135:6789/0 conn(0x7f9c840b1460 legacy=0x7f9c84141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.752+0000 7f9c8a392700 1 -- 172.21.15.40:0/1953156082 >> v1:172.21.15.40:6789/0 conn(0x7f9c840b1830 legacy=0x7f9c8413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.752+0000 7f9c8a392700 10 monclient: _finish_hunting 0 2023-09-30T00:47:56.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.752+0000 7f9c8a392700 1 monclient: found mon.noname-c 2023-09-30T00:47:56.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.752+0000 7f9c8a392700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:56.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.752+0000 7f9c8a392700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:47:56.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.752+0000 7f9c8a392700 1 -- 172.21.15.40:0/1953156082 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c840a74d0 con 0x7f9c840a5fb0 2023-09-30T00:47:56.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.751+0000 7f21e5466700 1 -- 172.21.15.40:0/25882841 wait complete. 2023-09-30T00:47:56.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.752+0000 7f21e5466700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.752+0000 7f21e5466700 10 monclient: monmap: 2023-09-30T00:47:56.768 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.768 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:56.769 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:56.769 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:56.769 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.769 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:56.769 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:56.769 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:56.769 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:56.769 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.753+0000 7f21e5466700 1 librados: starting msgr at 2023-09-30T00:47:56.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.753+0000 7f21e5466700 1 librados: starting objecter 2023-09-30T00:47:56.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.753+0000 7f21e5466700 1 Processor -- start 2023-09-30T00:47:56.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.754+0000 7f21e5466700 1 -- start start 2023-09-30T00:47:56.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.754+0000 7f21e5466700 1 librados: setting wanted keys 2023-09-30T00:47:56.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.754+0000 7f21e5466700 1 librados: calling monclient init 2023-09-30T00:47:56.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.754+0000 7f21e5466700 10 monclient: init 2023-09-30T00:47:56.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:47:56.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e006c580 0x7f21e014d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 10 monclient(hunting): picked mon.noname-b con 0x7f21e006c580 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:56.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21e006c950 0x7f21e015a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 10 monclient(hunting): picked mon.noname-a con 0x7f21e006c950 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:56.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21e013a4f0 0x7f21e015ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 10 monclient(hunting): picked mon.noname-c con 0x7f21e013a4f0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:56.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e013be80 con 0x7f21e006c950 2023-09-30T00:47:56.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e015b390 con 0x7f21e006c580 2023-09-30T00:47:56.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e013c310 con 0x7f21e013a4f0 2023-09-30T00:47:56.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.755+0000 7f21e5466700 10 monclient(hunting): authenticate will time out at 5338.056636s 2023-09-30T00:47:56.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.754+0000 7f4cf659c700 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2023-09-30T00:47:56.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.754+0000 7f4cf659c700 1 -- 172.21.15.40:0/4045507777 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cf806c950 msgr2=0x7f4cf80a95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.754+0000 7f4cf659c700 1 --2- 172.21.15.40:0/4045507777 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4cf806c950 0x7f4cf80a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.754+0000 7f4cf659c700 1 -- 172.21.15.40:0/4045507777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cf813a4f0 msgr2=0x7f4cf815a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.754+0000 7f4cf659c700 1 --2- 172.21.15.40:0/4045507777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cf813a4f0 0x7f4cf815a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.754+0000 7f4cf659c700 10 monclient: _finish_hunting 0 2023-09-30T00:47:56.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.754+0000 7f4cf659c700 1 monclient: found mon.noname-c 2023-09-30T00:47:56.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.754+0000 7f4cf659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:56.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.754+0000 7f4cf659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:47:56.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.754+0000 7f4cf659c700 1 -- 172.21.15.40:0/4045507777 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec009420 con 0x7f4cf806c580 2023-09-30T00:47:56.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.753+0000 7fa6ae653700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a813d8a0 0x7fa6a80a7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.753+0000 7fa6ae653700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a813d8a0 0x7fa6a80a7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43142/0 (socket says 172.21.15.40:43142) 2023-09-30T00:47:56.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.753+0000 7fa6ae653700 1 -- 172.21.15.40:0/2791422641 learned_addr learned my addr 172.21.15.40:0/2791422641 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:56.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.753+0000 7fa6ae653700 10 monclient(hunting): get_auth_request con 0x7fa6a813d8a0 auth_method 0 2023-09-30T00:47:56.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.753+0000 7fa6ae653700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.753+0000 7fa6ae653700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.753+0000 7fa6ae653700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.756+0000 7fa6ae653700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.756+0000 7fa6ae653700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.756+0000 7fa6ae653700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.760+0000 7f21df7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21e013a4f0 0x7f21e015ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.760+0000 7f4cf659c700 1 --2- 172.21.15.40:0/4045507777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4cf806c580 0x7f4cf813d5c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009860 tx=0x7f4cec01b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.760+0000 7f4ce67fc700 10 client.?.objecter ms_handle_connect 0x7f4cf806c580 2023-09-30T00:47:56.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.760+0000 7f4ce67fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:56.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.761+0000 7fa6ae653700 10 monclient(hunting): handle_auth_done global_id 4321 payload 824 2023-09-30T00:47:56.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.761+0000 7fa6ae653700 1 -- 172.21.15.40:0/2791422641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6a806c580 msgr2=0x7fa6a80a9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.761+0000 7fa6ae653700 1 --2- 172.21.15.40:0/2791422641 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa6a806c580 0x7fa6a80a9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.761+0000 7fa6ae653700 1 -- 172.21.15.40:0/2791422641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6a806c950 msgr2=0x7fa6a8118520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.761+0000 7fa6ae653700 1 --2- 172.21.15.40:0/2791422641 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6a806c950 0x7fa6a8118520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.761+0000 7fa6ae653700 10 monclient: _finish_hunting 0 2023-09-30T00:47:56.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.761+0000 7fa6ae653700 1 monclient: found mon.noname-b 2023-09-30T00:47:56.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.761+0000 7fa6ae653700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:56.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.761+0000 7fa6ae653700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:47:56.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.761+0000 7fa6ae653700 1 -- 172.21.15.40:0/2791422641 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa698009d80 con 0x7fa6a813d8a0 2023-09-30T00:47:56.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.762+0000 7f21df7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21e013a4f0 0x7f21e015ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36476/0 (socket says 172.21.15.40:36476) 2023-09-30T00:47:56.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.762+0000 7f21df7fe700 1 -- 172.21.15.40:0/3559502456 learned_addr learned my addr 172.21.15.40:0/3559502456 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:56.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.762+0000 7f21df7fe700 10 monclient(hunting): get_auth_request con 0x7f21e013a4f0 auth_method 0 2023-09-30T00:47:56.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.762+0000 7f21df7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.762+0000 7f21df7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.762+0000 7f21df7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.763+0000 7f21df7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.763+0000 7f21df7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.763+0000 7f21df7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.763+0000 7f21deffd700 1 --2- 172.21.15.40:0/3559502456 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e006c580 0x7f21e014d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.764+0000 7f9c8a392700 1 --2- 172.21.15.40:0/1953156082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c840a5fb0 0x7f9c8413b330 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c009e90 tx=0x7f9c7c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adf89b39814e0ac3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.764+0000 7f21deffd700 10 monclient(hunting): get_auth_request con 0x7f21e006c580 auth_method 0 2023-09-30T00:47:56.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.764+0000 7f21deffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.764+0000 7f21deffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.764+0000 7f21deffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.765+0000 7f21deffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.765+0000 7f21deffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.765+0000 7f21deffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.771+0000 7fa6ae653700 1 --2- 172.21.15.40:0/2791422641 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a813d8a0 0x7fa6a80a7f10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa69800c4d0 tx=0x7fa698018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.771+0000 7fa69dffb700 10 client.?.objecter ms_handle_connect 0x7fa6a813d8a0 2023-09-30T00:47:56.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.771+0000 7fa69dffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.769+0000 7f21deffd700 10 monclient(hunting): handle_auth_done global_id 4324 payload 824 2023-09-30T00:47:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.770+0000 7f21deffd700 1 -- 172.21.15.40:0/3559502456 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21e013a4f0 msgr2=0x7f21e015ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.770+0000 7f21deffd700 1 --2- 172.21.15.40:0/3559502456 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f21e013a4f0 0x7f21e015ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.770+0000 7f21deffd700 1 -- 172.21.15.40:0/3559502456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21e006c950 msgr2=0x7f21e015a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.770+0000 7f21deffd700 1 --2- 172.21.15.40:0/3559502456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21e006c950 0x7f21e015a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.770+0000 7f21deffd700 10 monclient: _finish_hunting 0 2023-09-30T00:47:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.770+0000 7f21deffd700 1 monclient: found mon.noname-b 2023-09-30T00:47:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.770+0000 7f21deffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:56.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.770+0000 7f21deffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:47:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.770+0000 7f21deffd700 1 -- 172.21.15.40:0/3559502456 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d0009d80 con 0x7f21e006c580 2023-09-30T00:47:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.772+0000 7f6d35413700 10 monclient: get_monmap_and_config 2023-09-30T00:47:56.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.772+0000 7f6d35413700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.772+0000 7f6d35413700 10 monclient: monmap: 2023-09-30T00:47:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:56.773891+0000 2023-09-30T00:47:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:56.773891+0000 2023-09-30T00:47:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:56.782 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:56.782 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.774+0000 7f6d35413700 1 Processor -- start 2023-09-30T00:47:56.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.774+0000 7f6d35413700 1 -- start start 2023-09-30T00:47:56.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.774+0000 7f6d35413700 10 monclient: init 2023-09-30T00:47:56.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.775+0000 7f6d35413700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.775+0000 7f6d35413700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-09-30T00:47:56.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.775+0000 7f6d35413700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d300b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:56.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.775+0000 7f6d35413700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d300b1830 0x7f6d300a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.775+0000 7f6d35413700 10 monclient(hunting): picked mon.noname-b con 0x7f6d300b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:56.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.775+0000 7f6d35413700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6d3013ac20 0x7f6d3013b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.775+0000 7f6d35413700 10 monclient(hunting): picked mon.noname-c con 0x7f6d3013ac20 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.775+0000 7f6d35413700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d300a5b20 con 0x7f6d300b1460 2023-09-30T00:47:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.775+0000 7f6d35413700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.775+0000 7f6d35413700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d30141e40 con 0x7f6d3013ac20 2023-09-30T00:47:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.775+0000 7f6d35413700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.775+0000 7f6d35413700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d300a64f0 con 0x7f6d300b1830 2023-09-30T00:47:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.775+0000 7f6d35413700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.775+0000 7f6d35413700 10 monclient(hunting): authenticate will time out at 5338.076725s 2023-09-30T00:47:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c88b8f700 1 -- 172.21.15.40:0/1953156082 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c00e070 con 0x7f9c840a5fb0 2023-09-30T00:47:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c88b8f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c88b8f700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c88b8f700 10 monclient: dump: 2023-09-30T00:47:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.785 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c88b8f700 10 monclient: _finish_auth 0 2023-09-30T00:47:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c88b8f700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c88b8f700 1 -- 172.21.15.40:0/1953156082 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7c00bc90 con 0x7f9c840a5fb0 2023-09-30T00:47:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c88b8f700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c88b8f700 1 -- 172.21.15.40:0/1953156082 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c012660 con 0x7f9c840a5fb0 2023-09-30T00:47:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c88b8f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c88b8f700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c88b8f700 10 monclient: dump: 2023-09-30T00:47:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.786 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.786 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.786 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.786 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.786 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.786 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.786 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c8bdf5700 5 monclient: authenticate success, global_id 4315 2023-09-30T00:47:56.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c8bdf5700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:56.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f9c8bdf5700 4 set_mon_vals no callback set 2023-09-30T00:47:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f6d2effd700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6d300b1460 0x7f6d30141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43308/0 (socket says 172.21.15.40:43308) 2023-09-30T00:47:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f6d2effd700 1 -- 172.21.15.40:0/2834582901 learned_addr learned my addr 172.21.15.40:0/2834582901 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.776+0000 7f21deffd700 1 --2- 172.21.15.40:0/3559502456 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e006c580 0x7f21e014d800 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f21d8007f60 tx=0x7f21d8009880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f21c6ffd700 10 client.?.objecter ms_handle_connect 0x7f21e006c580 2023-09-30T00:47:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f21c6ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f6d2e7fc700 1 --2- 172.21.15.40:0/2834582901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d300b1830 0x7f6d300a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f6d2e7fc700 10 monclient(hunting): get_auth_request con 0x7f6d300b1830 auth_method 0 2023-09-30T00:47:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f6d2e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f6d2e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f6d2e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f9c8bdf5700 10 monclient: shutdown 2023-09-30T00:47:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f9c8bdf5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/1953156082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c840a5fb0 msgr2=0x7f9c8413b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f9c8bdf5700 1 --2- 172.21.15.40:0/1953156082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c840a5fb0 0x7f9c8413b330 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c009e90 tx=0x7f9c7c009780 comp rx=0 tx=0).stop 2023-09-30T00:47:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/1953156082 shutdown_connections 2023-09-30T00:47:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f9c8bdf5700 1 --2- 172.21.15.40:0/1953156082 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c840a5fb0 0x7f9c8413b330 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/1953156082 >> 172.21.15.40:0/1953156082 conn(0x7f9c840631f0 msgr2=0x7f9c840635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.777+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/1953156082 shutdown_connections 2023-09-30T00:47:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.778+0000 7f6d2e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.778+0000 7f6d2e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.778+0000 7f6d2e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.779+0000 7fa69dffb700 1 -- 172.21.15.40:0/2791422641 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69801c040 con 0x7fa6a813d8a0 2023-09-30T00:47:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.779+0000 7fa69dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.779+0000 7fa69dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.779+0000 7fa69dffb700 10 monclient: dump: 2023-09-30T00:47:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.791 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.779+0000 7fa69dffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.779+0000 7fa69dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.779+0000 7fa6b00b6700 5 monclient: authenticate success, global_id 4321 2023-09-30T00:47:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.779+0000 7fa6b00b6700 10 monclient: _renew_subs 2023-09-30T00:47:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.779+0000 7fa6b00b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.779+0000 7fa6b00b6700 1 -- 172.21.15.40:0/2791422641 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a80a7670 con 0x7fa6a813d8a0 2023-09-30T00:47:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.780+0000 7fa6b00b6700 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.780+0000 7fa6b00b6700 10 monclient: _renew_subs 2023-09-30T00:47:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.780+0000 7fa6b00b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.780+0000 7fa6b00b6700 1 -- 172.21.15.40:0/2791422641 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a80a8740 con 0x7fa6a813d8a0 2023-09-30T00:47:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.780+0000 7fa6b00b6700 1 librados: init done 2023-09-30T00:47:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.781+0000 7fa6b00b6700 10 client.4321.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.781+0000 7fa6b00b6700 15 client.4321.objecter _session_command_op_assign -1 1 2023-09-30T00:47:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.781+0000 7fa6b00b6700 20 client.4321.objecter _get_session osd=-1 returning homeless 2023-09-30T00:47:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.781+0000 7fa6b00b6700 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.781+0000 7fa6b00b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.781+0000 7fa6b00b6700 1 -- 172.21.15.40:0/2791422641 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa6a8072b90 con 0x7fa6a813d8a0 2023-09-30T00:47:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.778+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/1953156082 wait complete. 2023-09-30T00:47:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.778+0000 7f9c8bdf5700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.778+0000 7f9c8bdf5700 10 monclient: monmap: 2023-09-30T00:47:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:56.794 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.779+0000 7f9c8bdf5700 1 librados: starting msgr at 2023-09-30T00:47:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.779+0000 7f9c8bdf5700 1 librados: starting objecter 2023-09-30T00:47:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.779+0000 7f9c8bdf5700 1 Processor -- start 2023-09-30T00:47:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.781+0000 7fa69dffb700 1 -- 172.21.15.40:0/2791422641 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69800c050 con 0x7fa6a813d8a0 2023-09-30T00:47:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.781+0000 7fa69dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.781+0000 7fa69dffb700 1 -- 172.21.15.40:0/2791422641 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa698010850 con 0x7fa6a813d8a0 2023-09-30T00:47:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.781+0000 7fa69dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.781+0000 7fa69dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.781+0000 7fa69dffb700 10 monclient: dump: 2023-09-30T00:47:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.796 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.782+0000 7f6d2e7fc700 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2023-09-30T00:47:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.782+0000 7f6d2e7fc700 1 -- 172.21.15.40:0/2834582901 >> v2:172.21.15.135:3300/0 conn(0x7f6d3013ac20 msgr2=0x7f6d3013b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.782+0000 7f6d2e7fc700 1 --2- 172.21.15.40:0/2834582901 >> v2:172.21.15.135:3300/0 conn(0x7f6d3013ac20 0x7f6d3013b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.782+0000 7f6d2e7fc700 1 -- 172.21.15.40:0/2834582901 >> v1:172.21.15.40:6789/0 conn(0x7f6d300b1460 legacy=0x7f6d30141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.782+0000 7f6d2e7fc700 10 monclient: _finish_hunting 0 2023-09-30T00:47:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.782+0000 7f6d2e7fc700 1 monclient: found mon.noname-b 2023-09-30T00:47:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.782+0000 7f6d2e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.782+0000 7f6d2e7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:47:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.782+0000 7f6d2e7fc700 1 -- 172.21.15.40:0/2834582901 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d300a71f0 con 0x7f6d300b1830 2023-09-30T00:47:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.782+0000 7fa69f7fe700 4 set_mon_vals no callback set 2023-09-30T00:47:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.783+0000 7f9c8bdf5700 1 -- start start 2023-09-30T00:47:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.783+0000 7f9c8bdf5700 1 librados: setting wanted keys 2023-09-30T00:47:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.783+0000 7f9c8bdf5700 1 librados: calling monclient init 2023-09-30T00:47:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.783+0000 7f9c8bdf5700 10 monclient: init 2023-09-30T00:47:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:47:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c840a5fb0 0x7f9c8406cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 10 monclient(hunting): picked mon.noname-a con 0x7f9c840a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c840b1460 0x7f9c841579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 10 monclient(hunting): picked mon.noname-b con 0x7f9c840b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9c840b1830 0x7f9c8415ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 10 monclient(hunting): picked mon.noname-c con 0x7f9c840b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8415a4b0 con 0x7f9c840a5fb0 2023-09-30T00:47:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c840aed00 con 0x7f9c840b1460 2023-09-30T00:47:56.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8415a700 con 0x7f9c840b1830 2023-09-30T00:47:56.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.784+0000 7f9c8bdf5700 10 monclient(hunting): authenticate will time out at 5338.085380s 2023-09-30T00:47:56.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.785+0000 7f9c89b91700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c840a5fb0 0x7f9c8406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.785+0000 7f9c89b91700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c840a5fb0 0x7f9c8406cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56520/0 (socket says 172.21.15.40:56520) 2023-09-30T00:47:56.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.785+0000 7f9c89b91700 1 -- 172.21.15.40:0/125250338 learned_addr learned my addr 172.21.15.40:0/125250338 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:56.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.785+0000 7f9c89b91700 10 monclient(hunting): get_auth_request con 0x7f9c840a5fb0 auth_method 0 2023-09-30T00:47:56.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.785+0000 7f9c89b91700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.785+0000 7f9c89b91700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.785+0000 7f9c89b91700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.786+0000 7f6d2e7fc700 1 --2- 172.21.15.40:0/2834582901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d300b1830 0x7f6d300a5fb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6d240098e0 tx=0x7f6d240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c600f0523742c94b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.787+0000 7f9c8a392700 1 --2- 172.21.15.40:0/125250338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9c840b1830 0x7f9c8415ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.787+0000 7f9c89b91700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.787+0000 7f9c89b91700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.788+0000 7f9c89b91700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.788+0000 7f9c8a392700 10 monclient(hunting): get_auth_request con 0x7f9c840b1830 auth_method 0 2023-09-30T00:47:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.788+0000 7f9c8a392700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.788+0000 7f9c8a392700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.788+0000 7f9c8a392700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.788+0000 7f9c8a392700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.788+0000 7f9c8a392700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.788+0000 7f9c8a392700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.793+0000 7f9c8a392700 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2023-09-30T00:47:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.793+0000 7f9c8a392700 1 -- 172.21.15.40:0/125250338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c840b1460 msgr2=0x7f9c841579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.793+0000 7f9c8a392700 1 --2- 172.21.15.40:0/125250338 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9c840b1460 0x7f9c841579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.793+0000 7f9c8a392700 1 -- 172.21.15.40:0/125250338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c840a5fb0 msgr2=0x7f9c8406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.793+0000 7f9c8a392700 1 --2- 172.21.15.40:0/125250338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c840a5fb0 0x7f9c8406cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.793+0000 7f9c8a392700 10 monclient: _finish_hunting 0 2023-09-30T00:47:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.793+0000 7f9c8a392700 1 monclient: found mon.noname-c 2023-09-30T00:47:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.793+0000 7f9c8a392700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.793+0000 7f9c8a392700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:47:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.793+0000 7f9c8a392700 1 -- 172.21.15.40:0/125250338 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7c009a20 con 0x7f9c840b1830 2023-09-30T00:47:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.794+0000 7f21c6ffd700 1 -- 172.21.15.40:0/3559502456 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d800ec40 con 0x7f21e006c580 2023-09-30T00:47:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.794+0000 7f21c6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.794+0000 7f21c6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.794+0000 7f21c6ffd700 10 monclient: dump: 2023-09-30T00:47:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.805 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.794+0000 7f21c6ffd700 10 monclient: _finish_auth 0 2023-09-30T00:47:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.794+0000 7f21c6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.794+0000 7f21c6ffd700 1 -- 172.21.15.40:0/3559502456 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d800eda0 con 0x7f21e006c580 2023-09-30T00:47:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.794+0000 7f21c6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.794+0000 7f21c6ffd700 1 -- 172.21.15.40:0/3559502456 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d8014670 con 0x7f21e006c580 2023-09-30T00:47:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.794+0000 7f21c6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.794+0000 7f21c6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.794+0000 7f21c6ffd700 10 monclient: dump: 2023-09-30T00:47:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.805 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.806 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.794+0000 7f21dd7fa700 4 set_mon_vals no callback set 2023-09-30T00:47:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.795+0000 7f982d08d700 10 monclient: get_monmap_and_config 2023-09-30T00:47:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.795+0000 7f982d08d700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.795+0000 7f982d08d700 10 monclient: monmap: 2023-09-30T00:47:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:56.796276+0000 2023-09-30T00:47:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:56.796276+0000 2023-09-30T00:47:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:56.810 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.795+0000 7f21e5466700 5 monclient: authenticate success, global_id 4324 2023-09-30T00:47:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.795+0000 7f21e5466700 10 monclient: _renew_subs 2023-09-30T00:47:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.795+0000 7f21e5466700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.795+0000 7f21e5466700 1 -- 172.21.15.40:0/3559502456 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e0137a90 con 0x7f21e006c580 2023-09-30T00:47:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.795+0000 7f21e5466700 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.795+0000 7f21e5466700 10 monclient: _renew_subs 2023-09-30T00:47:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.795+0000 7f21e5466700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.795+0000 7f21e5466700 1 -- 172.21.15.40:0/3559502456 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e013b840 con 0x7f21e006c580 2023-09-30T00:47:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.795+0000 7f21e5466700 1 librados: init done 2023-09-30T00:47:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.796+0000 7f21e5466700 10 client.4324.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.796+0000 7f21e5466700 15 client.4324.objecter _session_command_op_assign -1 1 2023-09-30T00:47:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.796+0000 7f21e5466700 20 client.4324.objecter _get_session osd=-1 returning homeless 2023-09-30T00:47:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.796+0000 7f21e5466700 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.796+0000 7f21e5466700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.796+0000 7f21e5466700 1 -- 172.21.15.40:0/3559502456 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21e0072b90 con 0x7f21e006c580 2023-09-30T00:47:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.796+0000 7f982d08d700 1 Processor -- start 2023-09-30T00:47:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.796+0000 7f982d08d700 1 -- start start 2023-09-30T00:47:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.796+0000 7f982d08d700 10 monclient: init 2023-09-30T00:47:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.796+0000 7f9c8a392700 1 --2- 172.21.15.40:0/125250338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9c840b1830 0x7f9c8415ac20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c016000 tx=0x7f9c7c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.796+0000 7f9c79ffb700 10 client.?.objecter ms_handle_connect 0x7f9c840b1830 2023-09-30T00:47:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.796+0000 7f9c79ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.797+0000 7f982d08d700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.797+0000 7f982d08d700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-09-30T00:47:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.797+0000 7f982d08d700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f98280b14a0 0x7f98280afe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.797+0000 7f982d08d700 10 monclient(hunting): picked mon.noname-c con 0x7f98280b14a0 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.797+0000 7f982d08d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f98280b1870 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.797+0000 7f982d08d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f982806dd20 0x7f982806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.797+0000 7f982d08d700 10 monclient(hunting): picked mon.noname-b con 0x7f982806dd20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.797+0000 7f982d08d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982813ff40 con 0x7f98280b1870 2023-09-30T00:47:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.797+0000 7f982d08d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.797+0000 7f982d08d700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f982806b030 con 0x7f98280b14a0 2023-09-30T00:47:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.797+0000 7f982d08d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.797+0000 7f982d08d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982806b310 con 0x7f982806dd20 2023-09-30T00:47:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.797+0000 7f982d08d700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.797+0000 7f982d08d700 10 monclient(hunting): authenticate will time out at 5338.098518s 2023-09-30T00:47:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.798+0000 7f982759e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f982806dd20 0x7f982806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.799+0000 7f982759e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f982806dd20 0x7f982806e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36506/0 (socket says 172.21.15.40:36506) 2023-09-30T00:47:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.799+0000 7f982759e700 1 -- 172.21.15.40:0/1667369228 learned_addr learned my addr 172.21.15.40:0/1667369228 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.799+0000 7f982759e700 10 monclient(hunting): get_auth_request con 0x7f982806dd20 auth_method 0 2023-09-30T00:47:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.799+0000 7f982759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.799+0000 7f982759e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.799+0000 7f982759e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.800+0000 7f982759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.800+0000 7f982759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.800+0000 7f982759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.802+0000 7f982759e700 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2023-09-30T00:47:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.802+0000 7f982759e700 1 -- 172.21.15.40:0/1667369228 >> v2:172.21.15.135:3300/0 conn(0x7f98280b14a0 msgr2=0x7f98280afe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:56.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.802+0000 7f982759e700 1 --2- 172.21.15.40:0/1667369228 >> v2:172.21.15.135:3300/0 conn(0x7f98280b14a0 0x7f98280afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.802+0000 7f982759e700 1 -- 172.21.15.40:0/1667369228 >> v1:172.21.15.40:6789/0 conn(0x7f98280b1870 legacy=0x7f9828145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.802+0000 7f982759e700 10 monclient: _finish_hunting 0 2023-09-30T00:47:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.802+0000 7f982759e700 1 monclient: found mon.noname-b 2023-09-30T00:47:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.802+0000 7f982759e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.802+0000 7f982759e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:47:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.802+0000 7f982759e700 1 -- 172.21.15.40:0/1667369228 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98280af8f0 con 0x7f982806dd20 2023-09-30T00:47:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.804+0000 7fa69dffb700 1 -- 172.21.15.40:0/2791422641 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa698010c80 con 0x7fa6a813d8a0 2023-09-30T00:47:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.805+0000 7fa69dffb700 1 --2- 172.21.15.40:0/2791422641 >> v2:172.21.15.135:6800/116426 conn(0x7fa68c068890 0x7fa68c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.805+0000 7fa6ade52700 1 --2- 172.21.15.40:0/2791422641 >> v2:172.21.15.135:6800/116426 conn(0x7fa68c068890 0x7fa68c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.805+0000 7fa6ade52700 10 monclient: get_auth_request con 0x7fa68c068890 auth_method 0 2023-09-30T00:47:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.805+0000 7fa69dffb700 10 client.4321.objecter ms_handle_connect 0x7fa68c068890 2023-09-30T00:47:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.805+0000 7fa6ade52700 1 --2- 172.21.15.40:0/2791422641 >> v2:172.21.15.135:6800/116426 conn(0x7fa68c068890 0x7fa68c06ad40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4006fd0 tx=0x7fa6a4006db0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.806+0000 7f982759e700 1 --2- 172.21.15.40:0/1667369228 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f982806dd20 0x7f982806e1a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9820009f20 tx=0x7f9820009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=631b7927b4f1e9d2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 1 -- 172.21.15.40:0/2791422641 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa698057840 con 0x7fa6a813d8a0 2023-09-30T00:47:56.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 10 client.4321.objecter ms_dispatch 0x7fa6a8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:56.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 3 client.4321.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:56.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 3 client.4321.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:56.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 10 client.4321.objecter checking command 1 2023-09-30T00:47:56.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 1 --2- 172.21.15.40:0/2791422641 >> v2:172.21.15.40:6812/117236 conn(0x7fa68c06e800 0x7fa68c070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 20 client.4321.objecter _get_session s=0x7fa68c06e150 osd=3 3 2023-09-30T00:47:56.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 20 client.4321.objecter put_session s=0x7fa68c06e150 osd=3 3 2023-09-30T00:47:56.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 15 client.4321.objecter _session_command_op_remove -1 1 2023-09-30T00:47:56.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 20 client.4321.objecter _get_session s=0x7fa68c06e150 osd=3 3 2023-09-30T00:47:56.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 20 client.4321.objecter get_session s=0x7fa68c06e150 osd=3 3 2023-09-30T00:47:56.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 15 client.4321.objecter _session_command_op_assign 3 1 2023-09-30T00:47:56.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 20 client.4321.objecter put_session s=0x7fa68c06e150 osd=3 4 2023-09-30T00:47:56.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 10 client.4321.objecter _send_command 1 2023-09-30T00:47:56.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 1 -- 172.21.15.40:0/2791422641 --> v2:172.21.15.40:6812/117236 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa68c071280 con 0x7fa68c06e800 2023-09-30T00:47:56.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.811+0000 7fa69dffb700 20 client.4321.objecter dump_active .. 0 homeless 2023-09-30T00:47:56.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.813+0000 7fa69dffb700 1 -- 172.21.15.40:0/2791422641 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa698053b80 con 0x7fa6a813d8a0 2023-09-30T00:47:56.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.813+0000 7fa69dffb700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:47:56.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.818+0000 7fa6ad651700 1 --2- 172.21.15.40:0/2791422641 >> v2:172.21.15.40:6812/117236 conn(0x7fa68c06e800 0x7fa68c070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.819+0000 7fa6ad651700 10 monclient: get_auth_request con 0x7fa68c06e800 auth_method 0 2023-09-30T00:47:56.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.822+0000 7fa69dffb700 10 client.4321.objecter ms_handle_connect 0x7fa68c06e800 2023-09-30T00:47:56.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.822+0000 7fa6ad651700 1 --2- 172.21.15.40:0/2791422641 >> v2:172.21.15.40:6812/117236 conn(0x7fa68c06e800 0x7fa68c070c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.842+0000 7f21c6ffd700 1 -- 172.21.15.40:0/3559502456 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f21d8014aa0 con 0x7f21e006c580 2023-09-30T00:47:56.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 1 --2- 172.21.15.40:0/3559502456 >> v2:172.21.15.135:6800/116426 conn(0x7f21cc068890 0x7f21cc06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 1 -- 172.21.15.40:0/3559502456 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f21d8085070 con 0x7f21e006c580 2023-09-30T00:47:56.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 10 client.4324.objecter ms_dispatch 0x7f21e0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:56.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 3 client.4324.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:56.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 3 client.4324.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:56.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21de7fc700 1 --2- 172.21.15.40:0/3559502456 >> v2:172.21.15.135:6800/116426 conn(0x7f21cc068890 0x7f21cc06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 10 client.4324.objecter checking command 1 2023-09-30T00:47:56.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21de7fc700 10 monclient: get_auth_request con 0x7f21cc068890 auth_method 0 2023-09-30T00:47:56.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 1 --2- 172.21.15.40:0/3559502456 >> v2:172.21.15.40:6804/117238 conn(0x7f21cc06e800 0x7f21cc070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 20 client.4324.objecter _get_session s=0x7f21cc06e150 osd=0 3 2023-09-30T00:47:56.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 20 client.4324.objecter put_session s=0x7f21cc06e150 osd=0 3 2023-09-30T00:47:56.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 15 client.4324.objecter _session_command_op_remove -1 1 2023-09-30T00:47:56.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 20 client.4324.objecter _get_session s=0x7f21cc06e150 osd=0 3 2023-09-30T00:47:56.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 20 client.4324.objecter get_session s=0x7f21cc06e150 osd=0 3 2023-09-30T00:47:56.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 15 client.4324.objecter _session_command_op_assign 0 1 2023-09-30T00:47:56.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 20 client.4324.objecter put_session s=0x7f21cc06e150 osd=0 4 2023-09-30T00:47:56.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 10 client.4324.objecter _send_command 1 2023-09-30T00:47:56.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 1 -- 172.21.15.40:0/3559502456 --> v2:172.21.15.40:6804/117238 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21cc071280 con 0x7f21cc06e800 2023-09-30T00:47:56.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 20 client.4324.objecter dump_active .. 0 homeless 2023-09-30T00:47:56.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 1 -- 172.21.15.40:0/3559502456 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21d8085440 con 0x7f21e006c580 2023-09-30T00:47:56.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.843+0000 7f21c6ffd700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:47:56.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.844+0000 7f21c6ffd700 10 client.4324.objecter ms_handle_connect 0x7f21cc068890 2023-09-30T00:47:56.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.844+0000 7f21de7fc700 1 --2- 172.21.15.40:0/3559502456 >> v2:172.21.15.135:6800/116426 conn(0x7f21cc068890 0x7f21cc06ad40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f21d4007fd0 tx=0x7f21d4007b20 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.846+0000 7f21df7fe700 1 --2- 172.21.15.40:0/3559502456 >> v2:172.21.15.40:6804/117238 conn(0x7f21cc06e800 0x7f21cc070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.846+0000 7f21df7fe700 10 monclient: get_auth_request con 0x7f21cc06e800 auth_method 0 2023-09-30T00:47:56.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.851+0000 7f21c6ffd700 10 client.4324.objecter ms_handle_connect 0x7f21cc06e800 2023-09-30T00:47:56.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.851+0000 7f21df7fe700 1 --2- 172.21.15.40:0/3559502456 >> v2:172.21.15.40:6804/117238 conn(0x7f21cc06e800 0x7f21cc070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:56.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.891+0000 7fa69dffb700 1 -- 172.21.15.40:0/2791422641 <== osd.3 v2:172.21.15.40:6812/117236 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa68c071280 con 0x7fa68c06e800 2023-09-30T00:47:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.891+0000 7fa69dffb700 10 client.4321.objecter ms_dispatch 0x7fa6a8001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:47:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.891+0000 7fa69dffb700 10 client.4321.objecter _finish_command 1 = system:0 2023-09-30T00:47:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.891+0000 7fa69dffb700 20 client.4321.objecter put_session s=0x7fa68c06e150 osd=3 4 2023-09-30T00:47:56.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.891+0000 7fa69dffb700 15 client.4321.objecter _session_command_op_remove 3 1 2023-09-30T00:47:56.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.908+0000 7fa6b00b6700 10 client.4321.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T00:47:56.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.908+0000 7fa6b00b6700 15 client.4321.objecter _session_command_op_assign -1 2 2023-09-30T00:47:56.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.908+0000 7fa6b00b6700 20 client.4321.objecter _get_session s=0x7fa68c06e150 osd=3 3 2023-09-30T00:47:56.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.908+0000 7fa6b00b6700 20 client.4321.objecter put_session s=0x7fa68c06e150 osd=3 3 2023-09-30T00:47:56.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.908+0000 7fa6b00b6700 20 client.4321.objecter _get_session s=0x7fa68c06e150 osd=3 3 2023-09-30T00:47:56.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.908+0000 7fa6b00b6700 15 client.4321.objecter _session_command_op_remove -1 2 2023-09-30T00:47:56.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.908+0000 7fa6b00b6700 20 client.4321.objecter get_session s=0x7fa68c06e150 osd=3 3 2023-09-30T00:47:56.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.908+0000 7fa6b00b6700 15 client.4321.objecter _session_command_op_assign 3 2 2023-09-30T00:47:56.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.908+0000 7fa6b00b6700 20 client.4321.objecter put_session s=0x7fa68c06e150 osd=3 4 2023-09-30T00:47:56.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.908+0000 7fa6b00b6700 10 client.4321.objecter _send_command 2 2023-09-30T00:47:56.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.908+0000 7fa6b00b6700 1 -- 172.21.15.40:0/2791422641 --> v2:172.21.15.40:6812/117236 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa6a8138570 con 0x7fa68c06e800 2023-09-30T00:47:56.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.913+0000 7f21c6ffd700 1 -- 172.21.15.40:0/3559502456 <== osd.0 v2:172.21.15.40:6804/117238 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f21cc071280 con 0x7f21cc06e800 2023-09-30T00:47:56.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.913+0000 7f21c6ffd700 10 client.4324.objecter ms_dispatch 0x7f21e0001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:47:56.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.913+0000 7f21c6ffd700 10 client.4324.objecter _finish_command 1 = system:0 2023-09-30T00:47:56.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.913+0000 7f21c6ffd700 20 client.4324.objecter put_session s=0x7f21cc06e150 osd=0 4 2023-09-30T00:47:56.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.913+0000 7f21c6ffd700 15 client.4324.objecter _session_command_op_remove 0 1 2023-09-30T00:47:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa69dffb700 1 -- 172.21.15.40:0/2791422641 <== osd.3 v2:172.21.15.40:6812/117236 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa6a8138570 con 0x7fa68c06e800 2023-09-30T00:47:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa69dffb700 10 client.4321.objecter ms_dispatch 0x7fa6a8001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:47:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa69dffb700 10 client.4321.objecter _finish_command 2 = system:0 2023-09-30T00:47:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa69dffb700 20 client.4321.objecter put_session s=0x7fa68c06e150 osd=3 4 2023-09-30T00:47:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa69dffb700 15 client.4321.objecter _session_command_op_remove 3 2 2023-09-30T00:47:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 10 librados: watch_flush enter 2023-09-30T00:47:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 10 librados: watch_flush exit 2023-09-30T00:47:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 10 client.4321.objecter close_session for osd.3 2023-09-30T00:47:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 1 -- 172.21.15.40:0/2791422641 >> v2:172.21.15.40:6812/117236 conn(0x7fa68c06e800 msgr2=0x7fa68c070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 1 --2- 172.21.15.40:0/2791422641 >> v2:172.21.15.40:6812/117236 conn(0x7fa68c06e800 0x7fa68c070c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 20 client.4321.objecter put_session s=0x7fa68c06e150 osd=3 1 2023-09-30T00:47:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 20 client.4321.objecter shutdown clearing up homeless session... 2023-09-30T00:47:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 10 client.4321.objecter successfully canceled tick 2023-09-30T00:47:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 1 -- 172.21.15.40:0/2791422641 >> v2:172.21.15.135:6800/116426 conn(0x7fa68c068890 msgr2=0x7fa68c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 1 --2- 172.21.15.40:0/2791422641 >> v2:172.21.15.135:6800/116426 conn(0x7fa68c068890 0x7fa68c06ad40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4006fd0 tx=0x7fa6a4006db0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 10 monclient: shutdown 2023-09-30T00:47:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 1 -- 172.21.15.40:0/2791422641 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a813d8a0 msgr2=0x7fa6a80a7f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 1 --2- 172.21.15.40:0/2791422641 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa6a813d8a0 0x7fa6a80a7f10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa69800c4d0 tx=0x7fa698018000 comp rx=0 tx=0).stop 2023-09-30T00:47:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6ae653700 1 -- 172.21.15.40:0/2791422641 reap_dead start 2023-09-30T00:47:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 1 -- 172.21.15.40:0/2791422641 shutdown_connections 2023-09-30T00:47:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.919+0000 7fa6b00b6700 1 -- 172.21.15.40:0/2791422641 >> 172.21.15.40:0/2791422641 conn(0x7fa6a8067bf0 msgr2=0x7fa6a813d070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.920+0000 7fa6b00b6700 1 -- 172.21.15.40:0/2791422641 shutdown_connections 2023-09-30T00:47:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.920+0000 7fa6b00b6700 1 -- 172.21.15.40:0/2791422641 wait complete. 2023-09-30T00:47:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.920+0000 7fa6b00b6700 1 librados: shutdown 2023-09-30T00:47:56.939 INFO:teuthology.orchestra.run.smithi040.stdout:85899345927 2023-09-30T00:47:56.940 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:47:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.940+0000 7f21e5466700 10 client.4324.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T00:47:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.940+0000 7f21e5466700 15 client.4324.objecter _session_command_op_assign -1 2 2023-09-30T00:47:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.940+0000 7f21e5466700 20 client.4324.objecter _get_session s=0x7f21cc06e150 osd=0 3 2023-09-30T00:47:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.940+0000 7f21e5466700 20 client.4324.objecter put_session s=0x7f21cc06e150 osd=0 3 2023-09-30T00:47:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.940+0000 7f21e5466700 20 client.4324.objecter _get_session s=0x7f21cc06e150 osd=0 3 2023-09-30T00:47:56.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.940+0000 7f21e5466700 15 client.4324.objecter _session_command_op_remove -1 2 2023-09-30T00:47:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.940+0000 7f21e5466700 20 client.4324.objecter get_session s=0x7f21cc06e150 osd=0 3 2023-09-30T00:47:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.940+0000 7f21e5466700 15 client.4324.objecter _session_command_op_assign 0 2 2023-09-30T00:47:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.940+0000 7f21e5466700 20 client.4324.objecter put_session s=0x7f21cc06e150 osd=0 4 2023-09-30T00:47:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.940+0000 7f21e5466700 10 client.4324.objecter _send_command 2 2023-09-30T00:47:56.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.940+0000 7f21e5466700 1 -- 172.21.15.40:0/3559502456 --> v2:172.21.15.40:6804/117238 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f21e006dec0 con 0x7f21cc06e800 2023-09-30T00:47:56.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63e3fff700 1 -- 172.21.15.40:0/3030148126 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc00b8b0 con 0x7f63ec13eb60 2023-09-30T00:47:56.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63e3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:56.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63e3fff700 10 monclient: dump: 2023-09-30T00:47:56.948 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.948 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.948 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.949 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63e3fff700 10 monclient: _finish_auth 0 2023-09-30T00:47:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63e3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63e3fff700 1 -- 172.21.15.40:0/3030148126 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63dc00ba10 con 0x7f63ec13eb60 2023-09-30T00:47:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63f3635700 5 monclient: authenticate success, global_id 4253 2023-09-30T00:47:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63f3635700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63e3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63f3635700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63e3fff700 1 -- 172.21.15.40:0/3030148126 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc00bdb0 con 0x7f63ec13eb60 2023-09-30T00:47:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63f3635700 4 set_mon_vals no callback set 2023-09-30T00:47:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63e3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.947+0000 7f63e3fff700 10 monclient: dump: 2023-09-30T00:47:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.951 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.948+0000 7f63f3635700 10 monclient: shutdown 2023-09-30T00:47:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.948+0000 7f63f3635700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.948+0000 7f63f3635700 1 -- 172.21.15.40:0/3030148126 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63ec13eb60 msgr2=0x7f63ec13ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.948+0000 7f63f3635700 1 --2- 172.21.15.40:0/3030148126 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63ec13eb60 0x7f63ec13ae40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f63dc0099b0 tx=0x7f63dc009250 comp rx=0 tx=0).stop 2023-09-30T00:47:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.948+0000 7f21c6ffd700 1 -- 172.21.15.40:0/3559502456 <== osd.0 v2:172.21.15.40:6804/117238 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f21e006dec0 con 0x7f21cc06e800 2023-09-30T00:47:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.948+0000 7f21c6ffd700 10 client.4324.objecter ms_dispatch 0x7f21e0001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:47:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.948+0000 7f21c6ffd700 10 client.4324.objecter _finish_command 2 = system:0 2023-09-30T00:47:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f21c6ffd700 20 client.4324.objecter put_session s=0x7f21cc06e150 osd=0 4 2023-09-30T00:47:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f21c6ffd700 15 client.4324.objecter _session_command_op_remove 0 2 2023-09-30T00:47:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f63f3635700 1 -- 172.21.15.40:0/3030148126 shutdown_connections 2023-09-30T00:47:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f63f3635700 1 --2- 172.21.15.40:0/3030148126 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63ec13eb60 0x7f63ec13ae40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f63f3635700 1 --2- 172.21.15.40:0/3030148126 >> v2:172.21.15.40:3300/0 conn(0x7f63ec13e1d0 0x7f63ec13e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f63f3635700 1 -- 172.21.15.40:0/3030148126 >> 172.21.15.40:0/3030148126 conn(0x7f63ec069080 msgr2=0x7f63ec0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59e77fe700 1 -- 172.21.15.40:0/3608755992 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc006af0 con 0x7f59e806c950 2023-09-30T00:47:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59e77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59e77fe700 10 monclient: dump: 2023-09-30T00:47:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.955 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59e77fe700 10 monclient: _finish_auth 0 2023-09-30T00:47:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59e77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:56.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59e77fe700 1 -- 172.21.15.40:0/3608755992 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59dc006c50 con 0x7f59e806c950 2023-09-30T00:47:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59eede8700 5 monclient: authenticate success, global_id 4247 2023-09-30T00:47:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59eede8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59e77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59eede8700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59e77fe700 1 -- 172.21.15.40:0/3608755992 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc00da60 con 0x7f59e806c950 2023-09-30T00:47:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59eede8700 4 set_mon_vals no callback set 2023-09-30T00:47:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59e77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59e77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f59e77fe700 10 monclient: dump: 2023-09-30T00:47:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.956 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.957 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.957 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.949+0000 7f21e5466700 10 librados: watch_flush enter 2023-09-30T00:47:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f21e5466700 10 librados: watch_flush exit 2023-09-30T00:47:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f21e5466700 10 client.4324.objecter close_session for osd.0 2023-09-30T00:47:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f21e5466700 1 -- 172.21.15.40:0/3559502456 >> v2:172.21.15.40:6804/117238 conn(0x7f21cc06e800 msgr2=0x7f21cc070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f21e5466700 1 --2- 172.21.15.40:0/3559502456 >> v2:172.21.15.40:6804/117238 conn(0x7f21cc06e800 0x7f21cc070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f21e5466700 20 client.4324.objecter put_session s=0x7f21cc06e150 osd=0 1 2023-09-30T00:47:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f21e5466700 20 client.4324.objecter shutdown clearing up homeless session... 2023-09-30T00:47:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f21e5466700 10 client.4324.objecter successfully canceled tick 2023-09-30T00:47:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f59eede8700 10 monclient: shutdown 2023-09-30T00:47:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f59eede8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f59eede8700 1 -- 172.21.15.40:0/3608755992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59e806c950 msgr2=0x7f59e806e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f59eede8700 1 --2- 172.21.15.40:0/3608755992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59e806c950 0x7f59e806e9f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00bb80 tx=0x7f59dc00b420 comp rx=0 tx=0).stop 2023-09-30T00:47:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f63f3635700 1 -- 172.21.15.40:0/3030148126 shutdown_connections 2023-09-30T00:47:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f63f3635700 1 -- 172.21.15.40:0/3030148126 wait complete. 2023-09-30T00:47:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f6d2dffb700 1 -- 172.21.15.40:0/2834582901 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d24009e80 con 0x7f6d300b1830 2023-09-30T00:47:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f6d2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f6d2dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f6d2dffb700 10 monclient: dump: 2023-09-30T00:47:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.961 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f6d2dffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f6d2dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f6d2dffb700 1 -- 172.21.15.40:0/2834582901 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2400b860 con 0x7f6d300b1830 2023-09-30T00:47:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f6d2dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f6d2dffb700 1 -- 172.21.15.40:0/2834582901 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2400bc00 con 0x7f6d300b1830 2023-09-30T00:47:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f6d2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f6d2dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f6d2dffb700 10 monclient: dump: 2023-09-30T00:47:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.962 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.963 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.951+0000 7f6d35413700 5 monclient: authenticate success, global_id 4262 2023-09-30T00:47:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.951+0000 7f6d35413700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.951+0000 7f6d35413700 4 set_mon_vals no callback set 2023-09-30T00:47:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.951+0000 7f6d35413700 10 monclient: shutdown 2023-09-30T00:47:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.951+0000 7f6d35413700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.951+0000 7f6d35413700 1 -- 172.21.15.40:0/2834582901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d300b1830 msgr2=0x7f6d300a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.951+0000 7f6d35413700 1 --2- 172.21.15.40:0/2834582901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d300b1830 0x7f6d300a5fb0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6d240098e0 tx=0x7f6d240091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f21e5466700 1 -- 172.21.15.40:0/3559502456 >> v2:172.21.15.135:6800/116426 conn(0x7f21cc068890 msgr2=0x7f21cc06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f21e5466700 1 --2- 172.21.15.40:0/3559502456 >> v2:172.21.15.135:6800/116426 conn(0x7f21cc068890 0x7f21cc06ad40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f21d4007fd0 tx=0x7f21d4007b20 comp rx=0 tx=0).stop 2023-09-30T00:47:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f21e5466700 10 monclient: shutdown 2023-09-30T00:47:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f21e5466700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f21e5466700 1 -- 172.21.15.40:0/3559502456 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e006c580 msgr2=0x7f21e014d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.950+0000 7f21e5466700 1 --2- 172.21.15.40:0/3559502456 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f21e006c580 0x7f21e014d800 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f21d8007f60 tx=0x7f21d8009880 comp rx=0 tx=0).stop 2023-09-30T00:47:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f21df7fe700 1 -- 172.21.15.40:0/3559502456 reap_dead start 2023-09-30T00:47:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f21e5466700 1 -- 172.21.15.40:0/3559502456 shutdown_connections 2023-09-30T00:47:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f21e5466700 1 -- 172.21.15.40:0/3559502456 >> 172.21.15.40:0/3559502456 conn(0x7f21e0067bf0 msgr2=0x7f21e006e230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.951+0000 7f63f3635700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.951+0000 7f63f3635700 10 monclient: monmap: 2023-09-30T00:47:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:56.966 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.951+0000 7f63f3635700 1 librados: starting msgr at 2023-09-30T00:47:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.951+0000 7f63f3635700 1 librados: starting objecter 2023-09-30T00:47:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.951+0000 7f63f3635700 1 Processor -- start 2023-09-30T00:47:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f63f3635700 1 -- start start 2023-09-30T00:47:56.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f63f3635700 1 librados: setting wanted keys 2023-09-30T00:47:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f63f3635700 1 librados: calling monclient init 2023-09-30T00:47:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f63f3635700 10 monclient: init 2023-09-30T00:47:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f6d35413700 1 -- 172.21.15.40:0/2834582901 shutdown_connections 2023-09-30T00:47:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f6d35413700 1 --2- 172.21.15.40:0/2834582901 >> v2:172.21.15.135:3300/0 conn(0x7f6d3013ac20 0x7f6d3013b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f6d35413700 1 --2- 172.21.15.40:0/2834582901 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d300b1830 0x7f6d300a5fb0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f6d35413700 1 -- 172.21.15.40:0/2834582901 >> 172.21.15.40:0/2834582901 conn(0x7f6d300631f0 msgr2=0x7f6d300635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f59eede8700 1 -- 172.21.15.40:0/3608755992 shutdown_connections 2023-09-30T00:47:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f59eede8700 1 --2- 172.21.15.40:0/3608755992 >> v2:172.21.15.135:3300/0 conn(0x7f59e813a4f0 0x7f59e813a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f59eede8700 1 --2- 172.21.15.40:0/3608755992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59e806c950 0x7f59e806e9f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.952+0000 7f59eede8700 1 -- 172.21.15.40:0/3608755992 >> 172.21.15.40:0/3608755992 conn(0x7f59e8067bf0 msgr2=0x7f59e80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:47:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f59eede8700 1 -- 172.21.15.40:0/3608755992 shutdown_connections 2023-09-30T00:47:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f9c79ffb700 1 -- 172.21.15.40:0/125250338 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c012d40 con 0x7f9c840b1830 2023-09-30T00:47:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f9c79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f9c79ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f9c79ffb700 10 monclient: dump: 2023-09-30T00:47:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.969 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.970 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f9c79ffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f9c79ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f9c79ffb700 1 -- 172.21.15.40:0/125250338 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7c021070 con 0x7f9c840b1830 2023-09-30T00:47:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f9c79ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:56.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f9c79ffb700 1 -- 172.21.15.40:0/125250338 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c026680 con 0x7f9c840b1830 2023-09-30T00:47:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f9c79ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f9c79ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f9c79ffb700 10 monclient: dump: 2023-09-30T00:47:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.972 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f9c8bdf5700 5 monclient: authenticate success, global_id 4265 2023-09-30T00:47:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f9c8bdf5700 10 monclient: _renew_subs 2023-09-30T00:47:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f9c8bdf5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/125250338 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c8406ea30 con 0x7f9c840b1830 2023-09-30T00:47:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f6d35413700 1 -- 172.21.15.40:0/2834582901 shutdown_connections 2023-09-30T00:47:56.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63ec0a5710 0x7f63ec0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 10 monclient(hunting): picked mon.noname-c con 0x7f63ec0a5710 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63ec13e1d0 0x7f63ec0ae120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 10 monclient(hunting): picked mon.noname-a con 0x7f63ec13e1d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63ec13eb60 0x7f63ec0ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 10 monclient(hunting): picked mon.noname-b con 0x7f63ec13eb60 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec0adcd0 con 0x7f63ec13e1d0 2023-09-30T00:47:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec0aeba0 con 0x7f63ec13eb60 2023-09-30T00:47:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63ec135f40 con 0x7f63ec0a5710 2023-09-30T00:47:56.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f63f3635700 10 monclient(hunting): authenticate will time out at 5338.254316s 2023-09-30T00:47:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4ce67fc700 1 -- 172.21.15.40:0/4045507777 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec0134d0 con 0x7f4cf806c580 2023-09-30T00:47:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4ce67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4ce67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4ce67fc700 10 monclient: dump: 2023-09-30T00:47:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.974 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.975 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4ce67fc700 10 monclient: _finish_auth 0 2023-09-30T00:47:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4ce67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4ce67fc700 1 -- 172.21.15.40:0/4045507777 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cec013630 con 0x7f4cf806c580 2023-09-30T00:47:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4cfcfba700 5 monclient: authenticate success, global_id 4256 2023-09-30T00:47:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4cfcfba700 10 monclient: _renew_subs 2023-09-30T00:47:56.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4cfcfba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:56.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4cfcfba700 1 -- 172.21.15.40:0/4045507777 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf815acb0 con 0x7f4cf806c580 2023-09-30T00:47:56.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4ce67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:56.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4ce67fc700 1 -- 172.21.15.40:0/4045507777 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec00f660 con 0x7f4cf806c580 2023-09-30T00:47:56.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4ce67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4cfcfba700 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:56.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4ce67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:56.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4ce67fc700 10 monclient: dump: 2023-09-30T00:47:56.976 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.976 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.976 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.976 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.976 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.977 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.977 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.977 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.977 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.977 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4cfcfba700 10 monclient: _renew_subs 2023-09-30T00:47:56.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4cfcfba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:56.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4cfcfba700 1 -- 172.21.15.40:0/4045507777 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf815b240 con 0x7f4cf806c580 2023-09-30T00:47:56.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4cfcfba700 1 librados: init done 2023-09-30T00:47:56.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f4cf4d99700 4 set_mon_vals no callback set 2023-09-30T00:47:56.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f59eede8700 1 -- 172.21.15.40:0/3608755992 wait complete. 2023-09-30T00:47:56.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f9c7bfff700 4 set_mon_vals no callback set 2023-09-30T00:47:56.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f9c8bdf5700 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:56.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f9c8bdf5700 10 monclient: _renew_subs 2023-09-30T00:47:56.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f9c8bdf5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:56.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/125250338 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c841580c0 con 0x7f9c840b1830 2023-09-30T00:47:56.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f9c8bdf5700 1 librados: init done 2023-09-30T00:47:56.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.953+0000 7f6d35413700 1 -- 172.21.15.40:0/2834582901 wait complete. 2023-09-30T00:47:56.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f6d35413700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f6d35413700 10 monclient: monmap: 2023-09-30T00:47:56.979 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.979 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:56.979 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:56.979 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:56.979 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.979 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:56.979 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:56.980 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f6d35413700 1 librados: starting msgr at 2023-09-30T00:47:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f6d35413700 1 librados: starting objecter 2023-09-30T00:47:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f6d35413700 1 Processor -- start 2023-09-30T00:47:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f63f1bd2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63ec13eb60 0x7f63ec0ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f63f1bd2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63ec13eb60 0x7f63ec0ae660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43212/0 (socket says 172.21.15.40:43212) 2023-09-30T00:47:56.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f63f1bd2700 1 -- 172.21.15.40:0/1526487877 learned_addr learned my addr 172.21.15.40:0/1526487877 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.954+0000 7f63f1bd2700 10 monclient(hunting): get_auth_request con 0x7f63ec13eb60 auth_method 0 2023-09-30T00:47:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f63f1bd2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f63f1bd2700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f63f1bd2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f4cfcfba700 10 client.4256.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f4cfcfba700 15 client.4256.objecter _session_command_op_assign -1 1 2023-09-30T00:47:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f4cfcfba700 20 client.4256.objecter _get_session osd=-1 returning homeless 2023-09-30T00:47:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f4cfcfba700 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:56.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f4cfcfba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f4cfcfba700 1 -- 172.21.15.40:0/4045507777 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4cf80a9d90 con 0x7f4cf806c580 2023-09-30T00:47:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f59eede8700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f59eede8700 10 monclient: monmap: 2023-09-30T00:47:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:56.982 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:56.983 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f59eede8700 1 librados: starting msgr at 2023-09-30T00:47:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f59eede8700 1 librados: starting objecter 2023-09-30T00:47:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f59eede8700 1 Processor -- start 2023-09-30T00:47:56.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f63f1bd2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f63f1bd2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f63f1bd2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f6d35413700 1 -- start start 2023-09-30T00:47:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f6d35413700 1 librados: setting wanted keys 2023-09-30T00:47:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f6d35413700 1 librados: calling monclient init 2023-09-30T00:47:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.955+0000 7f6d35413700 10 monclient: init 2023-09-30T00:47:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:47:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d300b1460 0x7f6d3006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 10 monclient(hunting): picked mon.noname-c con 0x7f6d300b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d300b1830 0x7f6d30155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 10 monclient(hunting): picked mon.noname-a con 0x7f6d300b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d3013ac20 0x7f6d3006c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 10 monclient(hunting): picked mon.noname-b con 0x7f6d3013ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:56.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d301562c0 con 0x7f6d300b1830 2023-09-30T00:47:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d30155e30 con 0x7f6d3013ac20 2023-09-30T00:47:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d30138670 con 0x7f6d300b1460 2023-09-30T00:47:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f6d35413700 10 monclient(hunting): authenticate will time out at 5338.257213s 2023-09-30T00:47:56.986 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f9c8bdf5700 10 client.4265.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f9c8bdf5700 15 client.4265.objecter _session_command_op_assign -1 1 2023-09-30T00:47:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f9c8bdf5700 20 client.4265.objecter _get_session osd=-1 returning homeless 2023-09-30T00:47:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f9c8bdf5700 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f9c8bdf5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/125250338 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9c84064a00 con 0x7f9c840b1830 2023-09-30T00:47:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f21e5466700 1 -- 172.21.15.40:0/3559502456 shutdown_connections 2023-09-30T00:47:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f21e5466700 1 -- 172.21.15.40:0/3559502456 wait complete. 2023-09-30T00:47:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f21e5466700 1 librados: shutdown 2023-09-30T00:47:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f59eede8700 1 -- start start 2023-09-30T00:47:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f59eede8700 1 librados: setting wanted keys 2023-09-30T00:47:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f59eede8700 1 librados: calling monclient init 2023-09-30T00:47:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.956+0000 7f59eede8700 10 monclient: init 2023-09-30T00:47:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:47:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59e806c580 0x7f59e8131cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 10 monclient(hunting): picked mon.noname-b con 0x7f59e806c580 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59e806c950 0x7f59e80a6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 10 monclient(hunting): picked mon.noname-a con 0x7f59e806c950 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59e813a4f0 0x7f59e80a7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 10 monclient(hunting): picked mon.noname-c con 0x7f59e813a4f0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e8133b40 con 0x7f59e806c950 2023-09-30T00:47:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e80a88e0 con 0x7f59e806c580 2023-09-30T00:47:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e81321f0 con 0x7f59e813a4f0 2023-09-30T00:47:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:56.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.957+0000 7f59eede8700 10 monclient(hunting): authenticate will time out at 5338.258489s 2023-09-30T00:47:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f9825d9b700 1 -- 172.21.15.40:0/1667369228 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982000e070 con 0x7f982806dd20 2023-09-30T00:47:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f9825d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f9825d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f9825d9b700 10 monclient: dump: 2023-09-30T00:47:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.991 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.992 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f9825d9b700 10 monclient: _finish_auth 0 2023-09-30T00:47:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f9825d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f9825d9b700 1 -- 172.21.15.40:0/1667369228 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982000ba80 con 0x7f982806dd20 2023-09-30T00:47:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f9825d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f9825d9b700 1 -- 172.21.15.40:0/1667369228 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98200092e0 con 0x7f982806dd20 2023-09-30T00:47:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f9825d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f9825d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:56.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f9825d9b700 10 monclient: dump: 2023-09-30T00:47:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:56.993 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f982d08d700 5 monclient: authenticate success, global_id 4268 2023-09-30T00:47:56.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f982d08d700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.958+0000 7f982d08d700 4 set_mon_vals no callback set 2023-09-30T00:47:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.959+0000 7f982d08d700 10 monclient: shutdown 2023-09-30T00:47:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.959+0000 7f982d08d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.959+0000 7f982d08d700 1 -- 172.21.15.40:0/1667369228 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f982806dd20 msgr2=0x7f982806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.959+0000 7f982d08d700 1 --2- 172.21.15.40:0/1667369228 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f982806dd20 0x7f982806e1a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9820009f20 tx=0x7f9820009840 comp rx=0 tx=0).stop 2023-09-30T00:47:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.959+0000 7f982d08d700 1 -- 172.21.15.40:0/1667369228 shutdown_connections 2023-09-30T00:47:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.959+0000 7f982d08d700 1 --2- 172.21.15.40:0/1667369228 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f982806dd20 0x7f982806e1a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.959+0000 7f982d08d700 1 --2- 172.21.15.40:0/1667369228 >> v2:172.21.15.135:3300/0 conn(0x7f98280b14a0 0x7f98280afe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.959+0000 7f982d08d700 1 -- 172.21.15.40:0/1667369228 >> 172.21.15.40:0/1667369228 conn(0x7f98280631f0 msgr2=0x7f98280635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.959+0000 7f982d08d700 1 -- 172.21.15.40:0/1667369228 shutdown_connections 2023-09-30T00:47:56.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.959+0000 7f982d08d700 1 -- 172.21.15.40:0/1667369228 wait complete. 2023-09-30T00:47:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.960+0000 7f982d08d700 10 monclient: build_initial_monmap 2023-09-30T00:47:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.960+0000 7f982d08d700 10 monclient: monmap: 2023-09-30T00:47:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:56.995 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:56.995 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:56.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f63f1bd2700 10 monclient(hunting): handle_auth_done global_id 4327 payload 824 2023-09-30T00:47:56.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f63f1bd2700 1 -- 172.21.15.40:0/1526487877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63ec0a5710 msgr2=0x7f63ec0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f63f1bd2700 1 --2- 172.21.15.40:0/1526487877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f63ec0a5710 0x7f63ec0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f63f1bd2700 1 -- 172.21.15.40:0/1526487877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63ec13e1d0 msgr2=0x7f63ec0ae120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:56.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f63f1bd2700 1 --2- 172.21.15.40:0/1526487877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63ec13e1d0 0x7f63ec0ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:56.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f63f1bd2700 10 monclient: _finish_hunting 0 2023-09-30T00:47:56.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f63f1bd2700 1 monclient: found mon.noname-b 2023-09-30T00:47:56.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f63f1bd2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:56.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f63f1bd2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:47:56.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f63f1bd2700 1 -- 172.21.15.40:0/1526487877 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63dc0094f0 con 0x7f63ec13eb60 2023-09-30T00:47:56.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f982d08d700 1 librados: starting msgr at 2023-09-30T00:47:56.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f982d08d700 1 librados: starting objecter 2023-09-30T00:47:56.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f982d08d700 1 Processor -- start 2023-09-30T00:47:56.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f6d2f7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d3013ac20 0x7f6d3006c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:56.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f982d08d700 1 -- start start 2023-09-30T00:47:56.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f982d08d700 1 librados: setting wanted keys 2023-09-30T00:47:56.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f982d08d700 1 librados: calling monclient init 2023-09-30T00:47:56.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f982d08d700 10 monclient: init 2023-09-30T00:47:56.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f6d2f7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d3013ac20 0x7f6d3006c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43228/0 (socket says 172.21.15.40:43228) 2023-09-30T00:47:56.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f6d2f7fe700 1 -- 172.21.15.40:0/3829480149 learned_addr learned my addr 172.21.15.40:0/3829480149 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:56.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f6d2f7fe700 10 monclient(hunting): get_auth_request con 0x7f6d3013ac20 auth_method 0 2023-09-30T00:47:56.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f6d2f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:56.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f6d2f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:56.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f6d2f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:56.999 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:47:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f982806dd20 0x7f982806bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 10 monclient(hunting): picked mon.noname-c con 0x7f982806dd20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98280b14a0 0x7f98281555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 10 monclient(hunting): picked mon.noname-a con 0x7f98280b14a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98280b1870 0x7f9828155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 10 monclient(hunting): picked mon.noname-b con 0x7f98280b1870 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982811d000 con 0x7f98280b14a0 2023-09-30T00:47:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9828151b50 con 0x7f98280b1870 2023-09-30T00:47:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98280b3f40 con 0x7f982806dd20 2023-09-30T00:47:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f982d08d700 10 monclient(hunting): authenticate will time out at 5338.263448s 2023-09-30T00:47:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.961+0000 7f59ecb84700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59e806c580 0x7f59e8131cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f59ecb84700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59e806c580 0x7f59e8131cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43242/0 (socket says 172.21.15.40:43242) 2023-09-30T00:47:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f59ecb84700 1 -- 172.21.15.40:0/1555501689 learned_addr learned my addr 172.21.15.40:0/1555501689 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f59ecb84700 10 monclient(hunting): get_auth_request con 0x7f59e806c580 auth_method 0 2023-09-30T00:47:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f59ecb84700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f59ecb84700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.962+0000 7f59ecb84700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.964+0000 7f63e17fa700 10 client.?.objecter ms_handle_connect 0x7f63ec13eb60 2023-09-30T00:47:57.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.964+0000 7f63e17fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:57.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.964+0000 7f59ecb84700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.964+0000 7f59ecb84700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.964+0000 7f59ecb84700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.964+0000 7f63f1bd2700 1 --2- 172.21.15.40:0/1526487877 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63ec13eb60 0x7f63ec0ae660 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f63dc0094c0 tx=0x7f63dc016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.965+0000 7f6d2f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.965+0000 7f6d2f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.965+0000 7f6d2f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.968+0000 7f59ecb84700 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2023-09-30T00:47:57.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.968+0000 7f59ecb84700 1 -- 172.21.15.40:0/1555501689 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59e813a4f0 msgr2=0x7f59e80a7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.968+0000 7f59ecb84700 1 --2- 172.21.15.40:0/1555501689 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f59e813a4f0 0x7f59e80a7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.968+0000 7f59ecb84700 1 -- 172.21.15.40:0/1555501689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59e806c950 msgr2=0x7f59e80a6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.968+0000 7f59ecb84700 1 --2- 172.21.15.40:0/1555501689 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59e806c950 0x7f59e80a6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.968+0000 7f59ecb84700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.968+0000 7f59ecb84700 1 monclient: found mon.noname-b 2023-09-30T00:47:57.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.968+0000 7f59ecb84700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.968+0000 7f59ecb84700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.968+0000 7f59ecb84700 1 -- 172.21.15.40:0/1555501689 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59dc00b6c0 con 0x7f59e806c580 2023-09-30T00:47:57.005 INFO:teuthology.orchestra.run.smithi040.stdout:85899345927 2023-09-30T00:47:57.006 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:47:57.006 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.975+0000 7f6d2f7fe700 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2023-09-30T00:47:57.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.975+0000 7f6d2f7fe700 1 -- 172.21.15.40:0/3829480149 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d300b1460 msgr2=0x7f6d3006dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.975+0000 7f6d2f7fe700 1 --2- 172.21.15.40:0/3829480149 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6d300b1460 0x7f6d3006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.975+0000 7f6d2f7fe700 1 -- 172.21.15.40:0/3829480149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d300b1830 msgr2=0x7f6d30155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.975+0000 7f6d2f7fe700 1 --2- 172.21.15.40:0/3829480149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6d300b1830 0x7f6d30155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.975+0000 7f6d2f7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.975+0000 7f6d2f7fe700 1 monclient: found mon.noname-b 2023-09-30T00:47:57.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.975+0000 7f6d2f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.975+0000 7f6d2f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.975+0000 7f6d2f7fe700 1 -- 172.21.15.40:0/3829480149 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d24009420 con 0x7f6d3013ac20 2023-09-30T00:47:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.977+0000 7f59ecb84700 1 --2- 172.21.15.40:0/1555501689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59e806c580 0x7f59e8131cb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f59e0001b80 tx=0x7f59e0006410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.977+0000 7f59e4ff9700 10 client.?.objecter ms_handle_connect 0x7f59e806c580 2023-09-30T00:47:57.008 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.977+0000 7f59e4ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63e17fa700 1 -- 172.21.15.40:0/1526487877 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc01c670 con 0x7f63ec13eb60 2023-09-30T00:47:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63e17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63e17fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63e17fa700 10 monclient: dump: 2023-09-30T00:47:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.009 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.010 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63e17fa700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63e17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63e17fa700 1 -- 172.21.15.40:0/1526487877 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63dc01c7d0 con 0x7f63ec13eb60 2023-09-30T00:47:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63f3635700 5 monclient: authenticate success, global_id 4327 2023-09-30T00:47:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63f3635700 10 monclient: _renew_subs 2023-09-30T00:47:57.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63f3635700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63f3635700 1 -- 172.21.15.40:0/1526487877 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63ec0ad680 con 0x7f63ec13eb60 2023-09-30T00:47:57.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63e17fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63e17fa700 1 -- 172.21.15.40:0/1526487877 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc01cb70 con 0x7f63ec13eb60 2023-09-30T00:47:57.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63e17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63e17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:57.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63e17fa700 10 monclient: dump: 2023-09-30T00:47:57.013 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.014 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.014 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.014 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.014 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.014 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.014 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.014 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.014 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.014 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63e37fe700 4 set_mon_vals no callback set 2023-09-30T00:47:57.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63f3635700 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63f3635700 10 monclient: _renew_subs 2023-09-30T00:47:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63f3635700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63f3635700 1 -- 172.21.15.40:0/1526487877 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63ec0ac3f0 con 0x7f63ec13eb60 2023-09-30T00:47:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.979+0000 7f63f3635700 1 librados: init done 2023-09-30T00:47:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.980+0000 7f63f3635700 10 client.4327.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.980+0000 7f63f3635700 15 client.4327.objecter _session_command_op_assign -1 1 2023-09-30T00:47:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.980+0000 7f63f3635700 20 client.4327.objecter _get_session osd=-1 returning homeless 2023-09-30T00:47:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.980+0000 7f63f3635700 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.980+0000 7f63f3635700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.980+0000 7f63f3635700 1 -- 172.21.15.40:0/1526487877 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f63ec0a5d30 con 0x7f63ec13eb60 2023-09-30T00:47:57.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.981+0000 7f6d2f7fe700 1 --2- 172.21.15.40:0/3829480149 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d3013ac20 0x7f6d3006c000 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800bc30 tx=0x7f6d28009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.981+0000 7f6d16ffd700 10 client.?.objecter ms_handle_connect 0x7f6d3013ac20 2023-09-30T00:47:57.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.981+0000 7f6d16ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.988+0000 7f59e4ff9700 1 -- 172.21.15.40:0/1555501689 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0002120 con 0x7f59e806c580 2023-09-30T00:47:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59e4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59e4ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59e4ff9700 10 monclient: dump: 2023-09-30T00:47:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.017 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.018 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59e4ff9700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59e4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59eede8700 5 monclient: authenticate success, global_id 4330 2023-09-30T00:47:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59eede8700 10 monclient: _renew_subs 2023-09-30T00:47:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59eede8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59eede8700 1 -- 172.21.15.40:0/1555501689 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e80a9520 con 0x7f59e806c580 2023-09-30T00:47:57.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59eede8700 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59eede8700 10 monclient: _renew_subs 2023-09-30T00:47:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59eede8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59eede8700 1 -- 172.21.15.40:0/1555501689 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e80a7b80 con 0x7f59e806c580 2023-09-30T00:47:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59eede8700 1 librados: init done 2023-09-30T00:47:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59eede8700 10 client.4330.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59eede8700 15 client.4330.objecter _session_command_op_assign -1 1 2023-09-30T00:47:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59eede8700 20 client.4330.objecter _get_session osd=-1 returning homeless 2023-09-30T00:47:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59eede8700 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.989+0000 7f59eede8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:56.990+0000 7f59eede8700 1 -- 172.21.15.40:0/1555501689 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f59e8133cb0 con 0x7f59e806c580 2023-09-30T00:47:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.000+0000 7f63e17fa700 1 -- 172.21.15.40:0/1526487877 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f63dc01a450 con 0x7f63ec13eb60 2023-09-30T00:47:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.001+0000 7f63e17fa700 1 --2- 172.21.15.40:0/1526487877 >> v2:172.21.15.135:6800/116426 conn(0x7f63d0068960 0x7f63d006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.001+0000 7f63f13d1700 1 --2- 172.21.15.40:0/1526487877 >> v2:172.21.15.135:6800/116426 conn(0x7f63d0068960 0x7f63d006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.001+0000 7f63f13d1700 10 monclient: get_auth_request con 0x7f63d0068960 auth_method 0 2023-09-30T00:47:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.002+0000 7f63e17fa700 10 client.4327.objecter ms_handle_connect 0x7f63d0068960 2023-09-30T00:47:57.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.002+0000 7f63f13d1700 1 --2- 172.21.15.40:0/1526487877 >> v2:172.21.15.135:6800/116426 conn(0x7f63d0068960 0x7f63d006ae10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f63e80097c0 tx=0x7f63e8006ce0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.004+0000 7f982759e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98280b1870 0x7f9828155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.005+0000 7f982759e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98280b1870 0x7f9828155ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43258/0 (socket says 172.21.15.40:43258) 2023-09-30T00:47:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.005+0000 7f982759e700 1 -- 172.21.15.40:0/1018353619 learned_addr learned my addr 172.21.15.40:0/1018353619 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.005+0000 7f982759e700 10 monclient(hunting): get_auth_request con 0x7f98280b1870 auth_method 0 2023-09-30T00:47:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.005+0000 7f982759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.005+0000 7f982759e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.005+0000 7f982759e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.007+0000 7f4ce67fc700 1 -- 172.21.15.40:0/4045507777 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f4cec00fa10 con 0x7f4cf806c580 2023-09-30T00:47:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.007+0000 7f982759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.007+0000 7f982759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.007+0000 7f982759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.007+0000 7f4ce67fc700 1 --2- 172.21.15.40:0/4045507777 >> v2:172.21.15.135:6800/116426 conn(0x7f4cd8068890 0x7f4cd806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.007+0000 7f4ce67fc700 1 -- 172.21.15.40:0/4045507777 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f4cec085080 con 0x7f4cf806c580 2023-09-30T00:47:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.007+0000 7f4ce67fc700 10 client.4256.objecter ms_dispatch 0x7f4cf8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.007+0000 7f4ce67fc700 3 client.4256.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.007+0000 7f4ce67fc700 3 client.4256.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.007+0000 7f4cf5d9b700 1 --2- 172.21.15.40:0/4045507777 >> v2:172.21.15.135:6800/116426 conn(0x7f4cd8068890 0x7f4cd806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4cf5d9b700 10 monclient: get_auth_request con 0x7f4cd8068890 auth_method 0 2023-09-30T00:47:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4ce67fc700 10 client.4256.objecter checking command 1 2023-09-30T00:47:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4ce67fc700 1 --2- 172.21.15.40:0/4045507777 >> v2:172.21.15.135:6809/117132 conn(0x7f4cd806e800 0x7f4cd8070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4ce67fc700 20 client.4256.objecter _get_session s=0x7f4cd806e150 osd=7 3 2023-09-30T00:47:57.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4ce67fc700 20 client.4256.objecter put_session s=0x7f4cd806e150 osd=7 3 2023-09-30T00:47:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4ce67fc700 15 client.4256.objecter _session_command_op_remove -1 1 2023-09-30T00:47:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4ce67fc700 20 client.4256.objecter _get_session s=0x7f4cd806e150 osd=7 3 2023-09-30T00:47:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4ce67fc700 20 client.4256.objecter get_session s=0x7f4cd806e150 osd=7 3 2023-09-30T00:47:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4ce67fc700 15 client.4256.objecter _session_command_op_assign 7 1 2023-09-30T00:47:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4ce67fc700 20 client.4256.objecter put_session s=0x7f4cd806e150 osd=7 4 2023-09-30T00:47:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4ce67fc700 10 client.4256.objecter _send_command 1 2023-09-30T00:47:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4ce67fc700 1 -- 172.21.15.40:0/4045507777 --> v2:172.21.15.135:6809/117132 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4cd8071280 con 0x7f4cd806e800 2023-09-30T00:47:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4ce67fc700 20 client.4256.objecter dump_active .. 0 homeless 2023-09-30T00:47:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4ce67fc700 10 client.4256.objecter ms_handle_connect 0x7f4cd8068890 2023-09-30T00:47:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4cf5d9b700 1 --2- 172.21.15.40:0/4045507777 >> v2:172.21.15.135:6800/116426 conn(0x7f4cd8068890 0x7f4cd806ad40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8006160 tx=0x7f4ce8005d40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.008+0000 7f4cf6d9d700 1 --2- 172.21.15.40:0/4045507777 >> v2:172.21.15.135:6809/117132 conn(0x7f4cd806e800 0x7f4cd8070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.009+0000 7f4cf6d9d700 10 monclient: get_auth_request con 0x7f4cd806e800 auth_method 0 2023-09-30T00:47:57.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.010+0000 7f982759e700 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2023-09-30T00:47:57.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.010+0000 7f982759e700 1 -- 172.21.15.40:0/1018353619 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f982806dd20 msgr2=0x7f982806bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.010+0000 7f982759e700 1 --2- 172.21.15.40:0/1018353619 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f982806dd20 0x7f982806bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.011+0000 7f982759e700 1 -- 172.21.15.40:0/1018353619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98280b14a0 msgr2=0x7f98281555a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.011+0000 7f982759e700 1 --2- 172.21.15.40:0/1018353619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98280b14a0 0x7f98281555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.011+0000 7f982759e700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.011+0000 7f982759e700 1 monclient: found mon.noname-b 2023-09-30T00:47:57.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.011+0000 7f982759e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.011+0000 7f982759e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.011+0000 7f982759e700 1 -- 172.21.15.40:0/1018353619 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9820009ae0 con 0x7f98280b1870 2023-09-30T00:47:57.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.011+0000 7f4ce67fc700 10 client.4256.objecter ms_handle_connect 0x7f4cd806e800 2023-09-30T00:47:57.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.011+0000 7f4cf6d9d700 1 --2- 172.21.15.40:0/4045507777 >> v2:172.21.15.135:6809/117132 conn(0x7f4cd806e800 0x7f4cd8070c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.018+0000 7f59e4ff9700 1 -- 172.21.15.40:0/1555501689 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e8133cb0 con 0x7f59e806c580 2023-09-30T00:47:57.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.018+0000 7f59e4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.018+0000 7f59e4ff9700 1 -- 172.21.15.40:0/1555501689 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e000d610 con 0x7f59e806c580 2023-09-30T00:47:57.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.018+0000 7f59e4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.018+0000 7f59e4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:57.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.018+0000 7f59e4ff9700 10 monclient: dump: 2023-09-30T00:47:57.028 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.028 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.029 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.029 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.029 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.029 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.029 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.029 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.029 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.029 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.018+0000 7f59e67fc700 4 set_mon_vals no callback set 2023-09-30T00:47:57.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d16ffd700 1 -- 172.21.15.40:0/3829480149 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d280092e0 con 0x7f6d3013ac20 2023-09-30T00:47:57.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d16ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d16ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:57.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d16ffd700 10 monclient: dump: 2023-09-30T00:47:57.031 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.031 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.031 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.031 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.031 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.032 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.032 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.032 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.032 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.032 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d16ffd700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d16ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d16ffd700 1 -- 172.21.15.40:0/3829480149 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2800c7a0 con 0x7f6d3013ac20 2023-09-30T00:47:57.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d35413700 5 monclient: authenticate success, global_id 4333 2023-09-30T00:47:57.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d35413700 10 monclient: _renew_subs 2023-09-30T00:47:57.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d35413700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d35413700 1 -- 172.21.15.40:0/3829480149 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d3006e270 con 0x7f6d3013ac20 2023-09-30T00:47:57.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d16ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d16ffd700 1 -- 172.21.15.40:0/3829480149 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2800cb80 con 0x7f6d3013ac20 2023-09-30T00:47:57.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d16ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d16ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:57.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d16ffd700 10 monclient: dump: 2023-09-30T00:47:57.033 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.033 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.033 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.033 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.033 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.033 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.034 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.034 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.034 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.034 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d35413700 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d35413700 10 monclient: _renew_subs 2023-09-30T00:47:57.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d35413700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d35413700 1 -- 172.21.15.40:0/3829480149 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d30155940 con 0x7f6d3013ac20 2023-09-30T00:47:57.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d35413700 1 librados: init done 2023-09-30T00:47:57.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.025+0000 7f6d2d7fa700 4 set_mon_vals no callback set 2023-09-30T00:47:57.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.026+0000 7f6d35413700 10 client.4333.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.026+0000 7f6d35413700 15 client.4333.objecter _session_command_op_assign -1 1 2023-09-30T00:47:57.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.026+0000 7f6d35413700 20 client.4333.objecter _get_session osd=-1 returning homeless 2023-09-30T00:47:57.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.026+0000 7f6d35413700 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.026+0000 7f6d35413700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.026+0000 7f6d35413700 1 -- 172.21.15.40:0/3829480149 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6d30155b90 con 0x7f6d3013ac20 2023-09-30T00:47:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.040+0000 7f59e4ff9700 1 -- 172.21.15.40:0/1555501689 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f59e000d770 con 0x7f59e806c580 2023-09-30T00:47:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.041+0000 7f59e4ff9700 1 --2- 172.21.15.40:0/1555501689 >> v2:172.21.15.135:6800/116426 conn(0x7f59d40688c0 0x7f59d406ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.041+0000 7f59e7fff700 1 --2- 172.21.15.40:0/1555501689 >> v2:172.21.15.135:6800/116426 conn(0x7f59d40688c0 0x7f59d406ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.041+0000 7f59e7fff700 10 monclient: get_auth_request con 0x7f59d40688c0 auth_method 0 2023-09-30T00:47:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.041+0000 7f59e4ff9700 10 client.4330.objecter ms_handle_connect 0x7f59d40688c0 2023-09-30T00:47:57.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.041+0000 7f59e7fff700 1 --2- 172.21.15.40:0/1555501689 >> v2:172.21.15.135:6800/116426 conn(0x7f59d40688c0 0x7f59d406ad70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f59e80aa1b0 tx=0x7f59dc01b000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.045+0000 7f59e4ff9700 1 -- 172.21.15.40:0/1555501689 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f59e00555e0 con 0x7f59e806c580 2023-09-30T00:47:57.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.045+0000 7f59e4ff9700 10 client.4330.objecter ms_dispatch 0x7f59e8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:57.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.045+0000 7f59e4ff9700 3 client.4330.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:57.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.045+0000 7f59e4ff9700 3 client.4330.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:57.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.045+0000 7f59e4ff9700 10 client.4330.objecter checking command 1 2023-09-30T00:47:57.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.045+0000 7f59e4ff9700 1 --2- 172.21.15.40:0/1555501689 >> v2:172.21.15.40:6800/117237 conn(0x7f59d406e830 0x7f59d4070c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.045+0000 7f59e4ff9700 20 client.4330.objecter _get_session s=0x7f59d406e180 osd=1 3 2023-09-30T00:47:57.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.046+0000 7f59e4ff9700 20 client.4330.objecter put_session s=0x7f59d406e180 osd=1 3 2023-09-30T00:47:57.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.046+0000 7f59e4ff9700 15 client.4330.objecter _session_command_op_remove -1 1 2023-09-30T00:47:57.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.046+0000 7f59e4ff9700 20 client.4330.objecter _get_session s=0x7f59d406e180 osd=1 3 2023-09-30T00:47:57.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.046+0000 7f59e4ff9700 20 client.4330.objecter get_session s=0x7f59d406e180 osd=1 3 2023-09-30T00:47:57.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.046+0000 7f59e4ff9700 15 client.4330.objecter _session_command_op_assign 1 1 2023-09-30T00:47:57.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.046+0000 7f59e4ff9700 20 client.4330.objecter put_session s=0x7f59d406e180 osd=1 4 2023-09-30T00:47:57.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.046+0000 7f59e4ff9700 10 client.4330.objecter _send_command 1 2023-09-30T00:47:57.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.046+0000 7f59e4ff9700 1 -- 172.21.15.40:0/1555501689 --> v2:172.21.15.40:6800/117237 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f59d40712b0 con 0x7f59d406e830 2023-09-30T00:47:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.046+0000 7f59e4ff9700 20 client.4330.objecter dump_active .. 0 homeless 2023-09-30T00:47:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.046+0000 7f982759e700 1 --2- 172.21.15.40:0/1018353619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98280b1870 0x7f9828155ae0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f982806b450 tx=0x7f9820018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.046+0000 7f980effd700 10 client.?.objecter ms_handle_connect 0x7f98280b1870 2023-09-30T00:47:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.046+0000 7f980effd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:57.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.046+0000 7f59ed385700 1 --2- 172.21.15.40:0/1555501689 >> v2:172.21.15.40:6800/117237 conn(0x7f59d406e830 0x7f59d4070c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.047+0000 7f59ed385700 10 monclient: get_auth_request con 0x7f59d406e830 auth_method 0 2023-09-30T00:47:57.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.047+0000 7f9c79ffb700 1 -- 172.21.15.40:0/125250338 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9c7c023030 con 0x7f9c840b1830 2023-09-30T00:47:57.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 1 --2- 172.21.15.40:0/125250338 >> v2:172.21.15.135:6800/116426 conn(0x7f9c68068890 0x7f9c6806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 1 -- 172.21.15.40:0/125250338 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9c7c086990 con 0x7f9c840b1830 2023-09-30T00:47:57.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 10 client.4265.objecter ms_dispatch 0x7f9c84001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:57.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 3 client.4265.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:57.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 3 client.4265.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:57.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c89b91700 1 --2- 172.21.15.40:0/125250338 >> v2:172.21.15.135:6800/116426 conn(0x7f9c68068890 0x7f9c6806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 10 client.4265.objecter checking command 1 2023-09-30T00:47:57.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c89b91700 10 monclient: get_auth_request con 0x7f9c68068890 auth_method 0 2023-09-30T00:47:57.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 1 --2- 172.21.15.40:0/125250338 >> v2:172.21.15.40:6808/117239 conn(0x7f9c6806e800 0x7f9c68070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 20 client.4265.objecter _get_session s=0x7f9c6806e150 osd=2 3 2023-09-30T00:47:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 20 client.4265.objecter put_session s=0x7f9c6806e150 osd=2 3 2023-09-30T00:47:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 15 client.4265.objecter _session_command_op_remove -1 1 2023-09-30T00:47:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 20 client.4265.objecter _get_session s=0x7f9c6806e150 osd=2 3 2023-09-30T00:47:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 20 client.4265.objecter get_session s=0x7f9c6806e150 osd=2 3 2023-09-30T00:47:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 15 client.4265.objecter _session_command_op_assign 2 1 2023-09-30T00:47:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 20 client.4265.objecter put_session s=0x7f9c6806e150 osd=2 4 2023-09-30T00:47:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 10 client.4265.objecter _send_command 1 2023-09-30T00:47:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 1 -- 172.21.15.40:0/125250338 --> v2:172.21.15.40:6808/117239 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c68071280 con 0x7f9c6806e800 2023-09-30T00:47:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 20 client.4265.objecter dump_active .. 0 homeless 2023-09-30T00:47:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c79ffb700 10 client.4265.objecter ms_handle_connect 0x7f9c68068890 2023-09-30T00:47:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.048+0000 7f9c89b91700 1 --2- 172.21.15.40:0/125250338 >> v2:172.21.15.135:6800/116426 conn(0x7f9c68068890 0x7f9c6806ad40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c740078a0 tx=0x7f9c74007830 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.049+0000 7f9c89390700 1 --2- 172.21.15.40:0/125250338 >> v2:172.21.15.40:6808/117239 conn(0x7f9c6806e800 0x7f9c68070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.049+0000 7f9c89390700 10 monclient: get_auth_request con 0x7f9c6806e800 auth_method 0 2023-09-30T00:47:57.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.051+0000 7f59e4ff9700 10 client.4330.objecter ms_handle_connect 0x7f59d406e830 2023-09-30T00:47:57.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.051+0000 7f59ed385700 1 --2- 172.21.15.40:0/1555501689 >> v2:172.21.15.40:6800/117237 conn(0x7f59d406e830 0x7f59d4070c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.056+0000 7f9c89390700 1 --2- 172.21.15.40:0/125250338 >> v2:172.21.15.40:6808/117239 conn(0x7f9c6806e800 0x7f9c68070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.056+0000 7f9c79ffb700 10 client.4265.objecter ms_handle_connect 0x7f9c6806e800 2023-09-30T00:47:57.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.066+0000 7f4ce67fc700 1 -- 172.21.15.40:0/4045507777 <== osd.7 v2:172.21.15.135:6809/117132 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f4cd8071280 con 0x7f4cd806e800 2023-09-30T00:47:57.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.066+0000 7f4ce67fc700 10 client.4256.objecter ms_dispatch 0x7f4cf8001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:47:57.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.066+0000 7f4ce67fc700 10 client.4256.objecter _finish_command 1 = system:0 2023-09-30T00:47:57.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.066+0000 7f4ce67fc700 20 client.4256.objecter put_session s=0x7f4cd806e150 osd=7 4 2023-09-30T00:47:57.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.066+0000 7f4ce67fc700 15 client.4256.objecter _session_command_op_remove 7 1 2023-09-30T00:47:57.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.078+0000 7f6d16ffd700 1 -- 172.21.15.40:0/3829480149 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6d28012070 con 0x7f6d3013ac20 2023-09-30T00:47:57.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 1 --2- 172.21.15.40:0/3829480149 >> v2:172.21.15.135:6800/116426 conn(0x7f6d1c068890 0x7f6d1c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 1 -- 172.21.15.40:0/3829480149 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6d28083e10 con 0x7f6d3013ac20 2023-09-30T00:47:57.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 10 client.4333.objecter ms_dispatch 0x7f6d30001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:57.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 3 client.4333.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:57.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 3 client.4333.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:57.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 10 client.4333.objecter checking command 1 2023-09-30T00:47:57.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d2effd700 1 --2- 172.21.15.40:0/3829480149 >> v2:172.21.15.135:6800/116426 conn(0x7f6d1c068890 0x7f6d1c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 1 --2- 172.21.15.40:0/3829480149 >> v2:172.21.15.135:6805/117131 conn(0x7f6d1c06e800 0x7f6d1c070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 20 client.4333.objecter _get_session s=0x7f6d1c06e150 osd=4 3 2023-09-30T00:47:57.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d2effd700 10 monclient: get_auth_request con 0x7f6d1c068890 auth_method 0 2023-09-30T00:47:57.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 20 client.4333.objecter put_session s=0x7f6d1c06e150 osd=4 3 2023-09-30T00:47:57.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 15 client.4333.objecter _session_command_op_remove -1 1 2023-09-30T00:47:57.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 20 client.4333.objecter _get_session s=0x7f6d1c06e150 osd=4 3 2023-09-30T00:47:57.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 20 client.4333.objecter get_session s=0x7f6d1c06e150 osd=4 3 2023-09-30T00:47:57.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 15 client.4333.objecter _session_command_op_assign 4 1 2023-09-30T00:47:57.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 20 client.4333.objecter put_session s=0x7f6d1c06e150 osd=4 4 2023-09-30T00:47:57.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 10 client.4333.objecter _send_command 1 2023-09-30T00:47:57.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 1 -- 172.21.15.40:0/3829480149 --> v2:172.21.15.135:6805/117131 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6d1c071280 con 0x7f6d1c06e800 2023-09-30T00:47:57.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.079+0000 7f6d16ffd700 20 client.4333.objecter dump_active .. 0 homeless 2023-09-30T00:47:57.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.080+0000 7f6d16ffd700 10 client.4333.objecter ms_handle_connect 0x7f6d1c068890 2023-09-30T00:47:57.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.080+0000 7f6d2effd700 1 --2- 172.21.15.40:0/3829480149 >> v2:172.21.15.135:6800/116426 conn(0x7f6d1c068890 0x7f6d1c06ad40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6d200016a0 tx=0x7f6d20007b30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.080+0000 7f6d2e7fc700 1 --2- 172.21.15.40:0/3829480149 >> v2:172.21.15.135:6805/117131 conn(0x7f6d1c06e800 0x7f6d1c070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.080+0000 7f6d2e7fc700 10 monclient: get_auth_request con 0x7f6d1c06e800 auth_method 0 2023-09-30T00:47:57.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.085+0000 7f6d16ffd700 10 client.4333.objecter ms_handle_connect 0x7f6d1c06e800 2023-09-30T00:47:57.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.085+0000 7f6d2e7fc700 1 --2- 172.21.15.40:0/3829480149 >> v2:172.21.15.135:6805/117131 conn(0x7f6d1c06e800 0x7f6d1c070c10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.100+0000 7f4cfcfba700 10 client.4256.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T00:47:57.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.100+0000 7f4cfcfba700 15 client.4256.objecter _session_command_op_assign -1 2 2023-09-30T00:47:57.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.100+0000 7f4cfcfba700 20 client.4256.objecter _get_session s=0x7f4cd806e150 osd=7 3 2023-09-30T00:47:57.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.100+0000 7f4cfcfba700 20 client.4256.objecter put_session s=0x7f4cd806e150 osd=7 3 2023-09-30T00:47:57.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.100+0000 7f4cfcfba700 20 client.4256.objecter _get_session s=0x7f4cd806e150 osd=7 3 2023-09-30T00:47:57.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.100+0000 7f4cfcfba700 15 client.4256.objecter _session_command_op_remove -1 2 2023-09-30T00:47:57.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.100+0000 7f4cfcfba700 20 client.4256.objecter get_session s=0x7f4cd806e150 osd=7 3 2023-09-30T00:47:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.100+0000 7f4cfcfba700 15 client.4256.objecter _session_command_op_assign 7 2 2023-09-30T00:47:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.100+0000 7f4cfcfba700 20 client.4256.objecter put_session s=0x7f4cd806e150 osd=7 4 2023-09-30T00:47:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.100+0000 7f4cfcfba700 10 client.4256.objecter _send_command 2 2023-09-30T00:47:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.100+0000 7f4cfcfba700 1 -- 172.21.15.40:0/4045507777 --> v2:172.21.15.135:6809/117132 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4cf815ae40 con 0x7f4cd806e800 2023-09-30T00:47:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.106+0000 7f1b481c1700 10 monclient: get_monmap_and_config 2023-09-30T00:47:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.106+0000 7f1b481c1700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.106+0000 7f1b481c1700 10 monclient: monmap: 2023-09-30T00:47:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:57.107176+0000 2023-09-30T00:47:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:57.107176+0000 2023-09-30T00:47:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:57.107 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:57.107 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.107+0000 7f1b481c1700 1 Processor -- start 2023-09-30T00:47:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.108+0000 7f1b481c1700 1 -- start start 2023-09-30T00:47:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.108+0000 7f1b481c1700 10 monclient: init 2023-09-30T00:47:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.109+0000 7f1b481c1700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.109+0000 7f1b481c1700 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-09-30T00:47:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.109+0000 7f1b481c1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1b400b16c0 0x7f1b400a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.109+0000 7f1b481c1700 10 monclient(hunting): picked mon.noname-a con 0x7f1b400b16c0 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.109+0000 7f1b481c1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b400a7610 0x7f1b400a7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.109+0000 7f1b481c1700 10 monclient(hunting): picked mon.noname-b con 0x7f1b400a7610 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.109+0000 7f1b481c1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b400a7f80 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:57.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.109+0000 7f1b481c1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400a5d00 con 0x7f1b400a7f80 2023-09-30T00:47:57.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.109+0000 7f1b481c1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.109+0000 7f1b481c1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b400b1a90 con 0x7f1b400b16c0 2023-09-30T00:47:57.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.109+0000 7f1b481c1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.109+0000 7f1b481c1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b400a5a00 con 0x7f1b400a7610 2023-09-30T00:47:57.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.109+0000 7f1b481c1700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.109+0000 7f1b481c1700 10 monclient(hunting): authenticate will time out at 5338.410485s 2023-09-30T00:47:57.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.110+0000 7f1b4675e700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1b400a7f80 0x7f1b400af400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43326/0 (socket says 172.21.15.40:43326) 2023-09-30T00:47:57.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.110+0000 7f1b4675e700 1 -- 172.21.15.40:0/1959819676 learned_addr learned my addr 172.21.15.40:0/1959819676 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:57.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.113+0000 7f4ce67fc700 1 -- 172.21.15.40:0/4045507777 <== osd.7 v2:172.21.15.135:6809/117132 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4cf815ae40 con 0x7f4cd806e800 2023-09-30T00:47:57.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.113+0000 7f4ce67fc700 10 client.4256.objecter ms_dispatch 0x7f4cf8001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:47:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.113+0000 7f4ce67fc700 10 client.4256.objecter _finish_command 2 = system:0 2023-09-30T00:47:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.113+0000 7f4ce67fc700 20 client.4256.objecter put_session s=0x7f4cd806e150 osd=7 4 2023-09-30T00:47:57.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.113+0000 7f4ce67fc700 15 client.4256.objecter _session_command_op_remove 7 2 2023-09-30T00:47:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 10 librados: watch_flush enter 2023-09-30T00:47:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 10 librados: watch_flush exit 2023-09-30T00:47:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 10 client.4256.objecter close_session for osd.7 2023-09-30T00:47:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 1 -- 172.21.15.40:0/4045507777 >> v2:172.21.15.135:6809/117132 conn(0x7f4cd806e800 msgr2=0x7f4cd8070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 1 --2- 172.21.15.40:0/4045507777 >> v2:172.21.15.135:6809/117132 conn(0x7f4cd806e800 0x7f4cd8070c10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 20 client.4256.objecter put_session s=0x7f4cd806e150 osd=7 1 2023-09-30T00:47:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 20 client.4256.objecter shutdown clearing up homeless session... 2023-09-30T00:47:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 10 client.4256.objecter successfully canceled tick 2023-09-30T00:47:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 1 -- 172.21.15.40:0/4045507777 >> v2:172.21.15.135:6800/116426 conn(0x7f4cd8068890 msgr2=0x7f4cd806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 1 --2- 172.21.15.40:0/4045507777 >> v2:172.21.15.135:6800/116426 conn(0x7f4cd8068890 0x7f4cd806ad40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8006160 tx=0x7f4ce8005d40 comp rx=0 tx=0).stop 2023-09-30T00:47:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 10 monclient: shutdown 2023-09-30T00:47:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 20 monclient: shutdown canceling and discarding version request 1 2023-09-30T00:47:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 1 -- 172.21.15.40:0/4045507777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4cf806c580 msgr2=0x7f4cf813d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cfcfba700 1 --2- 172.21.15.40:0/4045507777 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4cf806c580 0x7f4cf813d5c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009860 tx=0x7f4cec01b000 comp rx=0 tx=0).stop 2023-09-30T00:47:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f4cf6d9d700 1 -- 172.21.15.40:0/4045507777 reap_dead start 2023-09-30T00:47:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.114+0000 7f1b45f5d700 1 --2- 172.21.15.40:0/1959819676 >> v2:172.21.15.40:3300/0 conn(0x7f1b400b16c0 0x7f1b400a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.115+0000 7f4cfcfba700 1 -- 172.21.15.40:0/4045507777 shutdown_connections 2023-09-30T00:47:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.115+0000 7f4cfcfba700 1 -- 172.21.15.40:0/4045507777 >> 172.21.15.40:0/4045507777 conn(0x7f4cf8067bf0 msgr2=0x7f4cf806f060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.115+0000 7f1b45f5d700 10 monclient(hunting): get_auth_request con 0x7f1b400b16c0 auth_method 0 2023-09-30T00:47:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.115+0000 7f1b45f5d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.115+0000 7f1b45f5d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.115+0000 7f1b45f5d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.115+0000 7f1b45f5d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.115+0000 7f1b45f5d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.115+0000 7f1b45f5d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.119+0000 7f1b45f5d700 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2023-09-30T00:47:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.119+0000 7f1b45f5d700 1 -- 172.21.15.40:0/1959819676 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b400a7610 msgr2=0x7f1b400a7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.119+0000 7f1b45f5d700 1 --2- 172.21.15.40:0/1959819676 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b400a7610 0x7f1b400a7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.119+0000 7f1b45f5d700 1 -- 172.21.15.40:0/1959819676 >> v1:172.21.15.40:6789/0 conn(0x7f1b400a7f80 legacy=0x7f1b400af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.119+0000 7f1b45f5d700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.119+0000 7f1b45f5d700 1 monclient: found mon.noname-a 2023-09-30T00:47:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.119+0000 7f1b45f5d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.119+0000 7f1b45f5d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.119+0000 7f1b45f5d700 1 -- 172.21.15.40:0/1959819676 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b400a6c30 con 0x7f1b400b16c0 2023-09-30T00:47:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.121+0000 7f4cfcfba700 1 -- 172.21.15.40:0/4045507777 shutdown_connections 2023-09-30T00:47:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.121+0000 7f4cfcfba700 1 -- 172.21.15.40:0/4045507777 wait complete. 2023-09-30T00:47:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.121+0000 7f4cfcfba700 1 librados: shutdown 2023-09-30T00:47:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.121+0000 7f1b45f5d700 1 --2- 172.21.15.40:0/1959819676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b400b16c0 0x7f1b400a54c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1b300098e0 tx=0x7f1b300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7d0bf58d4727b2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.132+0000 7f9c79ffb700 1 -- 172.21.15.40:0/125250338 <== osd.2 v2:172.21.15.40:6808/117239 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f9c68071280 con 0x7f9c6806e800 2023-09-30T00:47:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.132+0000 7f9c79ffb700 10 client.4265.objecter ms_dispatch 0x7f9c84001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:47:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.132+0000 7f9c79ffb700 10 client.4265.objecter _finish_command 1 = system:0 2023-09-30T00:47:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.132+0000 7f9c79ffb700 20 client.4265.objecter put_session s=0x7f9c6806e150 osd=2 4 2023-09-30T00:47:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.132+0000 7f9c79ffb700 15 client.4265.objecter _session_command_op_remove 2 1 2023-09-30T00:47:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.128+0000 7f1b44f5b700 1 -- 172.21.15.40:0/1959819676 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30009e80 con 0x7f1b400b16c0 2023-09-30T00:47:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.128+0000 7f1b44f5b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.128+0000 7f1b44f5b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.128+0000 7f1b44f5b700 10 monclient: dump: 2023-09-30T00:47:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.135 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.128+0000 7f1b44f5b700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.128+0000 7f1b44f5b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.128+0000 7f1b481c1700 5 monclient: authenticate success, global_id 4299 2023-09-30T00:47:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.128+0000 7f1b481c1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.132+0000 7f1b44f5b700 1 -- 172.21.15.40:0/1959819676 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3000b760 con 0x7f1b400b16c0 2023-09-30T00:47:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.132+0000 7f1b44f5b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.132+0000 7f1b44f5b700 1 -- 172.21.15.40:0/1959819676 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3000bb00 con 0x7f1b400b16c0 2023-09-30T00:47:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.132+0000 7f1b44f5b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.132+0000 7f1b44f5b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.132+0000 7f1b44f5b700 10 monclient: dump: 2023-09-30T00:47:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.137 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.133+0000 7f1b481c1700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.133+0000 7f1b481c1700 4 set_mon_vals no callback set 2023-09-30T00:47:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.133+0000 7f1b481c1700 10 monclient: shutdown 2023-09-30T00:47:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.133+0000 7f1b481c1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.133+0000 7f1b481c1700 1 -- 172.21.15.40:0/1959819676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b400b16c0 msgr2=0x7f1b400a54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.133+0000 7f1b481c1700 1 --2- 172.21.15.40:0/1959819676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b400b16c0 0x7f1b400a54c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1b300098e0 tx=0x7f1b300091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.134+0000 7f1b481c1700 1 -- 172.21.15.40:0/1959819676 shutdown_connections 2023-09-30T00:47:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.134+0000 7f1b481c1700 1 --2- 172.21.15.40:0/1959819676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b400b16c0 0x7f1b400a54c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.134+0000 7f1b481c1700 1 --2- 172.21.15.40:0/1959819676 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b400a7610 0x7f1b400a7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.134+0000 7f1b481c1700 1 -- 172.21.15.40:0/1959819676 >> 172.21.15.40:0/1959819676 conn(0x7f1b4006f130 msgr2=0x7f1b400631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.135+0000 7f1b481c1700 1 -- 172.21.15.40:0/1959819676 shutdown_connections 2023-09-30T00:47:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.136+0000 7f1b481c1700 1 -- 172.21.15.40:0/1959819676 wait complete. 2023-09-30T00:47:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.137+0000 7f1b481c1700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.137+0000 7f1b481c1700 10 monclient: monmap: 2023-09-30T00:47:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:57.140 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.137+0000 7f1b481c1700 1 librados: starting msgr at 2023-09-30T00:47:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.137+0000 7f1b481c1700 1 librados: starting objecter 2023-09-30T00:47:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.138+0000 7f1b481c1700 1 Processor -- start 2023-09-30T00:47:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.138+0000 7f1b481c1700 1 -- start start 2023-09-30T00:47:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.138+0000 7f1b481c1700 1 librados: setting wanted keys 2023-09-30T00:47:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.138+0000 7f1b481c1700 1 librados: calling monclient init 2023-09-30T00:47:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.138+0000 7f1b481c1700 10 monclient: init 2023-09-30T00:47:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:47:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b400a7610 0x7f1b400a8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 10 monclient(hunting): picked mon.noname-a con 0x7f1b400a7610 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b400a7f80 0x7f1b4013ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 10 monclient(hunting): picked mon.noname-c con 0x7f1b400a7f80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b400b16c0 0x7f1b400a9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 10 monclient(hunting): picked mon.noname-b con 0x7f1b400b16c0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4013db30 con 0x7f1b400a7610 2023-09-30T00:47:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b400ab300 con 0x7f1b400b16c0 2023-09-30T00:47:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4013e470 con 0x7f1b400a7f80 2023-09-30T00:47:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.139+0000 7f1b481c1700 10 monclient(hunting): authenticate will time out at 5338.440193s 2023-09-30T00:47:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.144+0000 7f59e4ff9700 1 -- 172.21.15.40:0/1555501689 <== osd.1 v2:172.21.15.40:6800/117237 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f59d40712b0 con 0x7f59d406e830 2023-09-30T00:47:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.144+0000 7f59e4ff9700 10 client.4330.objecter ms_dispatch 0x7f59e8001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:47:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.144+0000 7f59e4ff9700 10 client.4330.objecter _finish_command 1 = system:0 2023-09-30T00:47:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.144+0000 7f59e4ff9700 20 client.4330.objecter put_session s=0x7f59d406e180 osd=1 4 2023-09-30T00:47:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.144+0000 7f59e4ff9700 15 client.4330.objecter _session_command_op_remove 1 1 2023-09-30T00:47:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.141+0000 7f1b45f5d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b400a7610 0x7f1b400a8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.141+0000 7f1b45f5d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b400a7610 0x7f1b400a8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56568/0 (socket says 172.21.15.40:56568) 2023-09-30T00:47:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.141+0000 7f1b45f5d700 1 -- 172.21.15.40:0/846249853 learned_addr learned my addr 172.21.15.40:0/846249853 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.141+0000 7f1b45f5d700 10 monclient(hunting): get_auth_request con 0x7f1b400a7610 auth_method 0 2023-09-30T00:47:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.141+0000 7f1b45f5d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.141+0000 7f1b45f5d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.141+0000 7f1b45f5d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.142+0000 7f1b45f5d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.142+0000 7f1b45f5d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.142+0000 7f1b45f5d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.146+0000 7f1b45f5d700 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2023-09-30T00:47:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.146+0000 7f1b45f5d700 1 -- 172.21.15.40:0/846249853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b400a7f80 msgr2=0x7f1b4013ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.146+0000 7f1b45f5d700 1 --2- 172.21.15.40:0/846249853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b400a7f80 0x7f1b4013ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.146+0000 7f1b45f5d700 1 -- 172.21.15.40:0/846249853 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b400b16c0 msgr2=0x7f1b400a9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.146+0000 7f1b45f5d700 1 --2- 172.21.15.40:0/846249853 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b400b16c0 0x7f1b400a9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.146+0000 7f1b45f5d700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.146+0000 7f1b45f5d700 1 monclient: found mon.noname-a 2023-09-30T00:47:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.146+0000 7f1b45f5d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.146+0000 7f1b45f5d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.146+0000 7f1b45f5d700 1 -- 172.21.15.40:0/846249853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b30009420 con 0x7f1b400a7610 2023-09-30T00:47:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.149+0000 7f1b45f5d700 1 --2- 172.21.15.40:0/846249853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b400a7610 0x7f1b400a8620 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1b300091e0 tx=0x7f1b30016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.149+0000 7f1b35ffb700 10 client.?.objecter ms_handle_connect 0x7f1b400a7610 2023-09-30T00:47:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.149+0000 7f1b35ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:57.151 INFO:teuthology.orchestra.run.smithi040.stdout:64424509446 2023-09-30T00:47:57.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T00:47:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b35ffb700 1 -- 172.21.15.40:0/846249853 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3000bec0 con 0x7f1b400a7610 2023-09-30T00:47:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b35ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b35ffb700 10 monclient: dump: 2023-09-30T00:47:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.157 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b35ffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b35ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b35ffb700 1 -- 172.21.15.40:0/846249853 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3000bae0 con 0x7f1b400a7610 2023-09-30T00:47:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b35ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b35ffb700 1 -- 172.21.15.40:0/846249853 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3001c6d0 con 0x7f1b400a7610 2023-09-30T00:47:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b35ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b35ffb700 10 monclient: dump: 2023-09-30T00:47:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.158 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b481c1700 5 monclient: authenticate success, global_id 4302 2023-09-30T00:47:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b377fe700 4 set_mon_vals no callback set 2023-09-30T00:47:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b481c1700 10 monclient: _renew_subs 2023-09-30T00:47:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b481c1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.154+0000 7f1b481c1700 1 -- 172.21.15.40:0/846249853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b400aa3c0 con 0x7f1b400a7610 2023-09-30T00:47:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.155+0000 7f1b481c1700 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.155+0000 7f1b481c1700 10 monclient: _renew_subs 2023-09-30T00:47:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.155+0000 7f1b481c1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.155+0000 7f1b481c1700 1 -- 172.21.15.40:0/846249853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b400aa7a0 con 0x7f1b400a7610 2023-09-30T00:47:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.155+0000 7f1b481c1700 1 librados: init done 2023-09-30T00:47:57.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.156+0000 7f1b481c1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.156+0000 7f1b481c1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.156+0000 7f1b481c1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:57.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.156+0000 7f1b481c1700 1 -- 172.21.15.40:0/846249853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b40064a00 con 0x7f1b400a7610 2023-09-30T00:47:57.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.158+0000 7f9c8bdf5700 10 client.4265.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T00:47:57.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f9c8bdf5700 15 client.4265.objecter _session_command_op_assign -1 2 2023-09-30T00:47:57.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f9c8bdf5700 20 client.4265.objecter _get_session s=0x7f9c6806e150 osd=2 3 2023-09-30T00:47:57.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f9c8bdf5700 20 client.4265.objecter put_session s=0x7f9c6806e150 osd=2 3 2023-09-30T00:47:57.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f9c8bdf5700 20 client.4265.objecter _get_session s=0x7f9c6806e150 osd=2 3 2023-09-30T00:47:57.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f9c8bdf5700 15 client.4265.objecter _session_command_op_remove -1 2 2023-09-30T00:47:57.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f9c8bdf5700 20 client.4265.objecter get_session s=0x7f9c6806e150 osd=2 3 2023-09-30T00:47:57.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f9c8bdf5700 15 client.4265.objecter _session_command_op_assign 2 2 2023-09-30T00:47:57.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f9c8bdf5700 20 client.4265.objecter put_session s=0x7f9c6806e150 osd=2 4 2023-09-30T00:47:57.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f9c8bdf5700 10 client.4265.objecter _send_command 2 2023-09-30T00:47:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/125250338 --> v2:172.21.15.40:6808/117239 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9c840a6410 con 0x7f9c6806e800 2023-09-30T00:47:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f6d16ffd700 1 -- 172.21.15.40:0/3829480149 <== osd.4 v2:172.21.15.135:6805/117131 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f6d1c071280 con 0x7f6d1c06e800 2023-09-30T00:47:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f6d16ffd700 10 client.4333.objecter ms_dispatch 0x7f6d30001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:47:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f6d16ffd700 10 client.4333.objecter _finish_command 1 = system:0 2023-09-30T00:47:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f6d16ffd700 20 client.4333.objecter put_session s=0x7f6d1c06e150 osd=4 4 2023-09-30T00:47:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.159+0000 7f6d16ffd700 15 client.4333.objecter _session_command_op_remove 4 1 2023-09-30T00:47:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 1 -- 172.21.15.40:0/1526487877 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f63dc057e40 con 0x7f63ec13eb60 2023-09-30T00:47:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 10 client.4327.objecter ms_dispatch 0x7f63ec001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 3 client.4327.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 3 client.4327.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 10 client.4327.objecter checking command 1 2023-09-30T00:47:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 1 --2- 172.21.15.40:0/1526487877 >> v2:172.21.15.135:6813/117133 conn(0x7f63d006e8d0 0x7f63d0070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 20 client.4327.objecter _get_session s=0x7f63d006e220 osd=6 3 2023-09-30T00:47:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 20 client.4327.objecter put_session s=0x7f63d006e220 osd=6 3 2023-09-30T00:47:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 15 client.4327.objecter _session_command_op_remove -1 1 2023-09-30T00:47:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 20 client.4327.objecter _get_session s=0x7f63d006e220 osd=6 3 2023-09-30T00:47:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 20 client.4327.objecter get_session s=0x7f63d006e220 osd=6 3 2023-09-30T00:47:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 15 client.4327.objecter _session_command_op_assign 6 1 2023-09-30T00:47:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 20 client.4327.objecter put_session s=0x7f63d006e220 osd=6 4 2023-09-30T00:47:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 10 client.4327.objecter _send_command 1 2023-09-30T00:47:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 1 -- 172.21.15.40:0/1526487877 --> v2:172.21.15.135:6813/117133 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63d0071350 con 0x7f63d006e8d0 2023-09-30T00:47:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 20 client.4327.objecter dump_active .. 0 homeless 2023-09-30T00:47:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 1 -- 172.21.15.40:0/1526487877 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63dc054180 con 0x7f63ec13eb60 2023-09-30T00:47:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63e17fa700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:47:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f63f0bd0700 1 --2- 172.21.15.40:0/1526487877 >> v2:172.21.15.135:6813/117133 conn(0x7f63d006e8d0 0x7f63d0070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f59e4ff9700 1 -- 172.21.15.40:0/1555501689 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59e0051920 con 0x7f59e806c580 2023-09-30T00:47:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.165+0000 7f59e4ff9700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:47:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.167+0000 7f6d16ffd700 1 -- 172.21.15.40:0/3829480149 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d280516d0 con 0x7f6d3013ac20 2023-09-30T00:47:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.167+0000 7f6d16ffd700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:47:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.167+0000 7f63f0bd0700 10 monclient: get_auth_request con 0x7f63d006e8d0 auth_method 0 2023-09-30T00:47:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.169+0000 7f9c79ffb700 1 -- 172.21.15.40:0/125250338 <== osd.2 v2:172.21.15.40:6808/117239 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9c840a6410 con 0x7f9c6806e800 2023-09-30T00:47:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.169+0000 7f9c79ffb700 10 client.4265.objecter ms_dispatch 0x7f9c84001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:47:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.169+0000 7f9c79ffb700 10 client.4265.objecter _finish_command 2 = system:0 2023-09-30T00:47:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.169+0000 7f9c79ffb700 20 client.4265.objecter put_session s=0x7f9c6806e150 osd=2 4 2023-09-30T00:47:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.169+0000 7f9c79ffb700 15 client.4265.objecter _session_command_op_remove 2 2 2023-09-30T00:47:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.169+0000 7f9c79ffb700 1 -- 172.21.15.40:0/125250338 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c7c054250 con 0x7f9c840b1830 2023-09-30T00:47:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.169+0000 7f9c79ffb700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:47:57.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.171+0000 7f63e17fa700 10 client.4327.objecter ms_handle_connect 0x7f63d006e8d0 2023-09-30T00:47:57.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.171+0000 7f63f0bd0700 1 --2- 172.21.15.40:0/1526487877 >> v2:172.21.15.135:6813/117133 conn(0x7f63d006e8d0 0x7f63d0070ce0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f980effd700 1 -- 172.21.15.40:0/1018353619 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98200032f0 con 0x7f98280b1870 2023-09-30T00:47:57.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f980effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f980effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:57.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f980effd700 10 monclient: dump: 2023-09-30T00:47:57.173 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.173 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.174 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f980effd700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f980effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f980effd700 1 -- 172.21.15.40:0/1018353619 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9820008e80 con 0x7f98280b1870 2023-09-30T00:47:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f980effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f980effd700 1 -- 172.21.15.40:0/1018353619 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98200266b0 con 0x7f98280b1870 2023-09-30T00:47:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f980effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f980effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f980effd700 10 monclient: dump: 2023-09-30T00:47:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.176 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f982d08d700 5 monclient: authenticate success, global_id 4336 2023-09-30T00:47:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f982d08d700 10 monclient: _renew_subs 2023-09-30T00:47:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f982d08d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f982d08d700 1 -- 172.21.15.40:0/1018353619 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9828151fe0 con 0x7f98280b1870 2023-09-30T00:47:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f982d08d700 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f982d08d700 10 monclient: _renew_subs 2023-09-30T00:47:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f982d08d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f982d08d700 1 -- 172.21.15.40:0/1018353619 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9828152500 con 0x7f98280b1870 2023-09-30T00:47:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.172+0000 7f982d08d700 1 librados: init done 2023-09-30T00:47:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.173+0000 7f982d08d700 10 client.4336.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.173+0000 7f982d08d700 15 client.4336.objecter _session_command_op_assign -1 1 2023-09-30T00:47:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.173+0000 7f982d08d700 20 client.4336.objecter _get_session osd=-1 returning homeless 2023-09-30T00:47:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.173+0000 7f982d08d700 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.173+0000 7f982d08d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.173+0000 7f982d08d700 1 -- 172.21.15.40:0/1018353619 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9828064a00 con 0x7f98280b1870 2023-09-30T00:47:57.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.179+0000 7f982559a700 4 set_mon_vals no callback set 2023-09-30T00:47:57.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8bdf5700 10 librados: watch_flush enter 2023-09-30T00:47:57.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8bdf5700 10 librados: watch_flush exit 2023-09-30T00:47:57.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8bdf5700 10 client.4265.objecter close_session for osd.2 2023-09-30T00:47:57.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/125250338 >> v2:172.21.15.40:6808/117239 conn(0x7f9c6806e800 msgr2=0x7f9c68070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8bdf5700 1 --2- 172.21.15.40:0/125250338 >> v2:172.21.15.40:6808/117239 conn(0x7f9c6806e800 0x7f9c68070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8bdf5700 20 client.4265.objecter put_session s=0x7f9c6806e150 osd=2 1 2023-09-30T00:47:57.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8bdf5700 20 client.4265.objecter shutdown clearing up homeless session... 2023-09-30T00:47:57.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8bdf5700 10 client.4265.objecter successfully canceled tick 2023-09-30T00:47:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/125250338 >> v2:172.21.15.135:6800/116426 conn(0x7f9c68068890 msgr2=0x7f9c6806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8bdf5700 1 --2- 172.21.15.40:0/125250338 >> v2:172.21.15.135:6800/116426 conn(0x7f9c68068890 0x7f9c6806ad40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c740078a0 tx=0x7f9c74007830 comp rx=0 tx=0).stop 2023-09-30T00:47:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8bdf5700 10 monclient: shutdown 2023-09-30T00:47:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8bdf5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/125250338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9c840b1830 msgr2=0x7f9c8415ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8bdf5700 1 --2- 172.21.15.40:0/125250338 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9c840b1830 0x7f9c8415ac20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c016000 tx=0x7f9c7c019000 comp rx=0 tx=0).stop 2023-09-30T00:47:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.180+0000 7f9c8a392700 1 -- 172.21.15.40:0/125250338 reap_dead start 2023-09-30T00:47:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.181+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/125250338 shutdown_connections 2023-09-30T00:47:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.181+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/125250338 >> 172.21.15.40:0/125250338 conn(0x7f9c840631f0 msgr2=0x7f9c840b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.182+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/125250338 shutdown_connections 2023-09-30T00:47:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.182+0000 7f9c8bdf5700 1 -- 172.21.15.40:0/125250338 wait complete. 2023-09-30T00:47:57.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.182+0000 7f9c8bdf5700 1 librados: shutdown 2023-09-30T00:47:57.191 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f6d35413700 10 client.4333.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T00:47:57.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f6d35413700 15 client.4333.objecter _session_command_op_assign -1 2 2023-09-30T00:47:57.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f6d35413700 20 client.4333.objecter _get_session s=0x7f6d1c06e150 osd=4 3 2023-09-30T00:47:57.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f6d35413700 20 client.4333.objecter put_session s=0x7f6d1c06e150 osd=4 3 2023-09-30T00:47:57.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f6d35413700 20 client.4333.objecter _get_session s=0x7f6d1c06e150 osd=4 3 2023-09-30T00:47:57.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f6d35413700 15 client.4333.objecter _session_command_op_remove -1 2 2023-09-30T00:47:57.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f6d35413700 20 client.4333.objecter get_session s=0x7f6d1c06e150 osd=4 3 2023-09-30T00:47:57.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f6d35413700 15 client.4333.objecter _session_command_op_assign 4 2 2023-09-30T00:47:57.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f6d35413700 20 client.4333.objecter put_session s=0x7f6d1c06e150 osd=4 4 2023-09-30T00:47:57.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f6d35413700 10 client.4333.objecter _send_command 2 2023-09-30T00:47:57.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f6d35413700 1 -- 172.21.15.40:0/3829480149 --> v2:172.21.15.135:6805/117131 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6d30064a00 con 0x7f6d1c06e800 2023-09-30T00:47:57.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f59eede8700 10 client.4330.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T00:47:57.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f59eede8700 15 client.4330.objecter _session_command_op_assign -1 2 2023-09-30T00:47:57.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f59eede8700 20 client.4330.objecter _get_session s=0x7f59d406e180 osd=1 3 2023-09-30T00:47:57.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f59eede8700 20 client.4330.objecter put_session s=0x7f59d406e180 osd=1 3 2023-09-30T00:47:57.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f59eede8700 20 client.4330.objecter _get_session s=0x7f59d406e180 osd=1 3 2023-09-30T00:47:57.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f59eede8700 15 client.4330.objecter _session_command_op_remove -1 2 2023-09-30T00:47:57.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f59eede8700 20 client.4330.objecter get_session s=0x7f59d406e180 osd=1 3 2023-09-30T00:47:57.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f59eede8700 15 client.4330.objecter _session_command_op_assign 1 2 2023-09-30T00:47:57.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f59eede8700 20 client.4330.objecter put_session s=0x7f59d406e180 osd=1 4 2023-09-30T00:47:57.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f59eede8700 10 client.4330.objecter _send_command 2 2023-09-30T00:47:57.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.191+0000 7f59eede8700 1 -- 172.21.15.40:0/1555501689 --> v2:172.21.15.40:6800/117237 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f59e80a8a20 con 0x7f59d406e830 2023-09-30T00:47:57.202 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.202+0000 7f980effd700 1 -- 172.21.15.40:0/1018353619 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9820003040 con 0x7f98280b1870 2023-09-30T00:47:57.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.202+0000 7f980effd700 1 --2- 172.21.15.40:0/1018353619 >> v2:172.21.15.135:6800/116426 conn(0x7f9814068890 0x7f981406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.203+0000 7f9826d9d700 1 --2- 172.21.15.40:0/1018353619 >> v2:172.21.15.135:6800/116426 conn(0x7f9814068890 0x7f981406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.203+0000 7f9826d9d700 10 monclient: get_auth_request con 0x7f9814068890 auth_method 0 2023-09-30T00:47:57.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.203+0000 7f980effd700 10 client.4336.objecter ms_handle_connect 0x7f9814068890 2023-09-30T00:47:57.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.203+0000 7f9826d9d700 1 --2- 172.21.15.40:0/1018353619 >> v2:172.21.15.135:6800/116426 conn(0x7f9814068890 0x7f981406ad40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f981c006fd0 tx=0x7f981c006db0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.203+0000 7f6d16ffd700 1 -- 172.21.15.40:0/3829480149 <== osd.4 v2:172.21.15.135:6805/117131 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6d30064a00 con 0x7f6d1c06e800 2023-09-30T00:47:57.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.203+0000 7f6d16ffd700 10 client.4333.objecter ms_dispatch 0x7f6d30001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:47:57.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.203+0000 7f6d16ffd700 10 client.4333.objecter _finish_command 2 = system:0 2023-09-30T00:47:57.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.203+0000 7f6d16ffd700 20 client.4333.objecter put_session s=0x7f6d1c06e150 osd=4 4 2023-09-30T00:47:57.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.203+0000 7f6d16ffd700 15 client.4333.objecter _session_command_op_remove 4 2 2023-09-30T00:47:57.206 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d35413700 10 librados: watch_flush enter 2023-09-30T00:47:57.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d35413700 10 librados: watch_flush exit 2023-09-30T00:47:57.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d35413700 10 client.4333.objecter close_session for osd.4 2023-09-30T00:47:57.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d35413700 1 -- 172.21.15.40:0/3829480149 >> v2:172.21.15.135:6805/117131 conn(0x7f6d1c06e800 msgr2=0x7f6d1c070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d35413700 1 --2- 172.21.15.40:0/3829480149 >> v2:172.21.15.135:6805/117131 conn(0x7f6d1c06e800 0x7f6d1c070c10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d35413700 20 client.4333.objecter put_session s=0x7f6d1c06e150 osd=4 1 2023-09-30T00:47:57.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d35413700 20 client.4333.objecter shutdown clearing up homeless session... 2023-09-30T00:47:57.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d35413700 10 client.4333.objecter successfully canceled tick 2023-09-30T00:47:57.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d35413700 1 -- 172.21.15.40:0/3829480149 >> v2:172.21.15.135:6800/116426 conn(0x7f6d1c068890 msgr2=0x7f6d1c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d35413700 1 --2- 172.21.15.40:0/3829480149 >> v2:172.21.15.135:6800/116426 conn(0x7f6d1c068890 0x7f6d1c06ad40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6d200016a0 tx=0x7f6d20007b30 comp rx=0 tx=0).stop 2023-09-30T00:47:57.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d35413700 10 monclient: shutdown 2023-09-30T00:47:57.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d35413700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:57.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d35413700 1 -- 172.21.15.40:0/3829480149 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d3013ac20 msgr2=0x7f6d3006c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d35413700 1 --2- 172.21.15.40:0/3829480149 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6d3013ac20 0x7f6d3006c000 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800bc30 tx=0x7f6d28009840 comp rx=0 tx=0).stop 2023-09-30T00:47:57.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.206+0000 7f6d2f7fe700 1 -- 172.21.15.40:0/3829480149 reap_dead start 2023-09-30T00:47:57.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f6d35413700 1 -- 172.21.15.40:0/3829480149 shutdown_connections 2023-09-30T00:47:57.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f6d35413700 1 -- 172.21.15.40:0/3829480149 >> 172.21.15.40:0/3829480149 conn(0x7f6d300631f0 msgr2=0x7f6d30139d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:57.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 1 -- 172.21.15.40:0/1018353619 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9820057e90 con 0x7f98280b1870 2023-09-30T00:47:57.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 10 client.4336.objecter ms_dispatch 0x7f9828001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:57.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 3 client.4336.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:57.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 3 client.4336.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:57.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 10 client.4336.objecter checking command 1 2023-09-30T00:47:57.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 1 --2- 172.21.15.40:0/1018353619 >> v2:172.21.15.135:6801/117130 conn(0x7f981406e800 0x7f9814070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 20 client.4336.objecter _get_session s=0x7f981406e150 osd=5 3 2023-09-30T00:47:57.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 20 client.4336.objecter put_session s=0x7f981406e150 osd=5 3 2023-09-30T00:47:57.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 15 client.4336.objecter _session_command_op_remove -1 1 2023-09-30T00:47:57.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 20 client.4336.objecter _get_session s=0x7f981406e150 osd=5 3 2023-09-30T00:47:57.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 20 client.4336.objecter get_session s=0x7f981406e150 osd=5 3 2023-09-30T00:47:57.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 15 client.4336.objecter _session_command_op_assign 5 1 2023-09-30T00:47:57.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 20 client.4336.objecter put_session s=0x7f981406e150 osd=5 4 2023-09-30T00:47:57.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 10 client.4336.objecter _send_command 1 2023-09-30T00:47:57.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 1 -- 172.21.15.40:0/1018353619 --> v2:172.21.15.135:6801/117130 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9814071280 con 0x7f981406e800 2023-09-30T00:47:57.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.207+0000 7f980effd700 20 client.4336.objecter dump_active .. 0 homeless 2023-09-30T00:47:57.211 INFO:teuthology.orchestra.run.smithi040.stdout:85899345926 2023-09-30T00:47:57.212 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:47:57.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.208+0000 7f980effd700 1 -- 172.21.15.40:0/1018353619 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9820053ad0 con 0x7f98280b1870 2023-09-30T00:47:57.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.208+0000 7f980effd700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:47:57.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.208+0000 7f6d35413700 1 -- 172.21.15.40:0/3829480149 shutdown_connections 2023-09-30T00:47:57.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.208+0000 7f6d35413700 1 -- 172.21.15.40:0/3829480149 wait complete. 2023-09-30T00:47:57.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.208+0000 7f6d35413700 1 librados: shutdown 2023-09-30T00:47:57.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.208+0000 7f982659c700 1 --2- 172.21.15.40:0/1018353619 >> v2:172.21.15.135:6801/117130 conn(0x7f981406e800 0x7f9814070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.208+0000 7f982659c700 10 monclient: get_auth_request con 0x7f981406e800 auth_method 0 2023-09-30T00:47:57.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.213+0000 7f982659c700 1 --2- 172.21.15.40:0/1018353619 >> v2:172.21.15.135:6801/117130 conn(0x7f981406e800 0x7f9814070c10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.213+0000 7f980effd700 10 client.4336.objecter ms_handle_connect 0x7f981406e800 2023-09-30T00:47:57.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.213+0000 7f96a61d1700 10 monclient: get_monmap_and_config 2023-09-30T00:47:57.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.213+0000 7f96a61d1700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.213+0000 7f96a61d1700 10 monclient: monmap: 2023-09-30T00:47:57.215 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.215 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.215 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:57.214118+0000 2023-09-30T00:47:57.215 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:57.214118+0000 2023-09-30T00:47:57.215 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.215 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.215 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:57.215 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:57.216 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:57.216 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:57.216 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:57.216 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.214+0000 7f96a61d1700 1 Processor -- start 2023-09-30T00:47:57.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.214+0000 7f96a61d1700 1 -- start start 2023-09-30T00:47:57.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.214+0000 7f96a61d1700 10 monclient: init 2023-09-30T00:47:57.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.215+0000 7f96a61d1700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.215+0000 7f96a61d1700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2023-09-30T00:47:57.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.215+0000 7f96a61d1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f96a006ad50 0x7f96a006b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.215+0000 7f96a61d1700 10 monclient(hunting): picked mon.noname-a con 0x7f96a006ad50 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:57.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.215+0000 7f96a61d1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96a006beb0 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:57.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.215+0000 7f96a61d1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96a01432a0 0x7f96a006ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.215+0000 7f96a61d1700 10 monclient(hunting): picked mon.noname-b con 0x7f96a01432a0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.215+0000 7f96a61d1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a0142e10 con 0x7f96a006beb0 2023-09-30T00:47:57.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.215+0000 7f96a61d1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.215+0000 7f96a61d1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96a0142b90 con 0x7f96a006ad50 2023-09-30T00:47:57.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.215+0000 7f96a61d1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.215+0000 7f96a61d1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a006cef0 con 0x7f96a01432a0 2023-09-30T00:47:57.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.215+0000 7f96a61d1700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.215+0000 7f96a61d1700 10 monclient(hunting): authenticate will time out at 5338.516278s 2023-09-30T00:47:57.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.219+0000 7f59e4ff9700 1 -- 172.21.15.40:0/1555501689 <== osd.1 v2:172.21.15.40:6800/117237 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f59e80a8a20 con 0x7f59d406e830 2023-09-30T00:47:57.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.219+0000 7f59e4ff9700 10 client.4330.objecter ms_dispatch 0x7f59e8001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:47:57.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.219+0000 7f59e4ff9700 10 client.4330.objecter _finish_command 2 = system:0 2023-09-30T00:47:57.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.219+0000 7f59e4ff9700 20 client.4330.objecter put_session s=0x7f59d406e180 osd=1 4 2023-09-30T00:47:57.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.219+0000 7f59e4ff9700 15 client.4330.objecter _session_command_op_remove 1 2 2023-09-30T00:47:57.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.220+0000 7f969ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96a01432a0 0x7f96a006ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.221+0000 7f969ffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96a01432a0 0x7f96a006ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36570/0 (socket says 172.21.15.40:36570) 2023-09-30T00:47:57.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.221+0000 7f969ffff700 1 -- 172.21.15.40:0/3370129519 learned_addr learned my addr 172.21.15.40:0/3370129519 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f969ffff700 10 monclient(hunting): get_auth_request con 0x7f96a01432a0 auth_method 0 2023-09-30T00:47:57.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f969ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f969ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f969ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.221+0000 7f59eede8700 10 librados: watch_flush enter 2023-09-30T00:47:57.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f59eede8700 10 librados: watch_flush exit 2023-09-30T00:47:57.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f59eede8700 10 client.4330.objecter close_session for osd.1 2023-09-30T00:47:57.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f59eede8700 1 -- 172.21.15.40:0/1555501689 >> v2:172.21.15.40:6800/117237 conn(0x7f59d406e830 msgr2=0x7f59d4070c40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f59eede8700 1 --2- 172.21.15.40:0/1555501689 >> v2:172.21.15.40:6800/117237 conn(0x7f59d406e830 0x7f59d4070c40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f59eede8700 20 client.4330.objecter put_session s=0x7f59d406e180 osd=1 1 2023-09-30T00:47:57.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f59eede8700 20 client.4330.objecter shutdown clearing up homeless session... 2023-09-30T00:47:57.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f59eede8700 10 client.4330.objecter successfully canceled tick 2023-09-30T00:47:57.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f59eede8700 1 -- 172.21.15.40:0/1555501689 >> v2:172.21.15.135:6800/116426 conn(0x7f59d40688c0 msgr2=0x7f59d406ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f59eede8700 1 --2- 172.21.15.40:0/1555501689 >> v2:172.21.15.135:6800/116426 conn(0x7f59d40688c0 0x7f59d406ad70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f59e80aa1b0 tx=0x7f59dc01b000 comp rx=0 tx=0).stop 2023-09-30T00:47:57.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f59eede8700 10 monclient: shutdown 2023-09-30T00:47:57.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f59eede8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:57.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f59eede8700 1 -- 172.21.15.40:0/1555501689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59e806c580 msgr2=0x7f59e8131cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f59eede8700 1 --2- 172.21.15.40:0/1555501689 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59e806c580 0x7f59e8131cb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f59e0001b80 tx=0x7f59e0006410 comp rx=0 tx=0).stop 2023-09-30T00:47:57.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.222+0000 7f59ed385700 1 -- 172.21.15.40:0/1555501689 reap_dead start 2023-09-30T00:47:57.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.223+0000 7f969ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.223+0000 7f969ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.223+0000 7f969ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.223+0000 7f59eede8700 1 -- 172.21.15.40:0/1555501689 shutdown_connections 2023-09-30T00:47:57.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.223+0000 7f59eede8700 1 -- 172.21.15.40:0/1555501689 >> 172.21.15.40:0/1555501689 conn(0x7f59e8067bf0 msgr2=0x7f59e80ad980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:57.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.225+0000 7f59eede8700 1 -- 172.21.15.40:0/1555501689 shutdown_connections 2023-09-30T00:47:57.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.225+0000 7f59eede8700 1 -- 172.21.15.40:0/1555501689 wait complete. 2023-09-30T00:47:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.226+0000 7f59eede8700 1 librados: shutdown 2023-09-30T00:47:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.226+0000 7f969ffff700 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2023-09-30T00:47:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.226+0000 7f969ffff700 1 -- 172.21.15.40:0/3370129519 >> v2:172.21.15.40:3300/0 conn(0x7f96a006ad50 msgr2=0x7f96a006b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.226+0000 7f969ffff700 1 --2- 172.21.15.40:0/3370129519 >> v2:172.21.15.40:3300/0 conn(0x7f96a006ad50 0x7f96a006b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.226+0000 7f969ffff700 1 -- 172.21.15.40:0/3370129519 >> v1:172.21.15.40:6789/0 conn(0x7f96a006beb0 legacy=0x7f96a006e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.226+0000 7f969ffff700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.226+0000 7f969ffff700 1 monclient: found mon.noname-b 2023-09-30T00:47:57.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.226+0000 7f969ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.226+0000 7f969ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.226+0000 7f969ffff700 1 -- 172.21.15.40:0/3370129519 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a006f350 con 0x7f96a01432a0 2023-09-30T00:47:57.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.228+0000 7f969ffff700 1 --2- 172.21.15.40:0/3370129519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96a01432a0 0x7f96a006ee10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9698009f20 tx=0x7f9698009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f86f0f4ea4f3aace server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.230 INFO:teuthology.orchestra.run.smithi040.stdout:64424509446 2023-09-30T00:47:57.230 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T00:47:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.232+0000 7f969e7fc700 1 -- 172.21.15.40:0/3370129519 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969800e070 con 0x7f96a01432a0 2023-09-30T00:47:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.232+0000 7f969e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.232+0000 7f969e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.232+0000 7f969e7fc700 10 monclient: dump: 2023-09-30T00:47:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.233 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.234 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.232+0000 7f969e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.232+0000 7f969e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.232+0000 7f96a61d1700 5 monclient: authenticate success, global_id 4271 2023-09-30T00:47:57.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.232+0000 7f96a61d1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:57.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.233+0000 7f63e17fa700 1 -- 172.21.15.40:0/1526487877 <== osd.6 v2:172.21.15.135:6813/117133 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f63d0071350 con 0x7f63d006e8d0 2023-09-30T00:47:57.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.233+0000 7f63e17fa700 10 client.4327.objecter ms_dispatch 0x7f63ec001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:47:57.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.233+0000 7f63e17fa700 10 client.4327.objecter _finish_command 1 = system:0 2023-09-30T00:47:57.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.233+0000 7f63e17fa700 20 client.4327.objecter put_session s=0x7f63d006e220 osd=6 4 2023-09-30T00:47:57.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.233+0000 7f63e17fa700 15 client.4327.objecter _session_command_op_remove 6 1 2023-09-30T00:47:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.235+0000 7f969e7fc700 1 -- 172.21.15.40:0/3370129519 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969800bad0 con 0x7f96a01432a0 2023-09-30T00:47:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.235+0000 7f969e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.235+0000 7f969e7fc700 1 -- 172.21.15.40:0/3370129519 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96980092e0 con 0x7f96a01432a0 2023-09-30T00:47:57.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.235+0000 7f969e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.235+0000 7f969e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.235+0000 7f969e7fc700 10 monclient: dump: 2023-09-30T00:47:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.242 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.243 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.235+0000 7f96a61d1700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.235+0000 7f96a61d1700 4 set_mon_vals no callback set 2023-09-30T00:47:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.236+0000 7f96a61d1700 10 monclient: shutdown 2023-09-30T00:47:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.236+0000 7f96a61d1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.236+0000 7f96a61d1700 1 -- 172.21.15.40:0/3370129519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96a01432a0 msgr2=0x7f96a006ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.236+0000 7f96a61d1700 1 --2- 172.21.15.40:0/3370129519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96a01432a0 0x7f96a006ee10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9698009f20 tx=0x7f9698009840 comp rx=0 tx=0).stop 2023-09-30T00:47:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.236+0000 7f96a61d1700 1 -- 172.21.15.40:0/3370129519 shutdown_connections 2023-09-30T00:47:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.236+0000 7f96a61d1700 1 --2- 172.21.15.40:0/3370129519 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96a01432a0 0x7f96a006ee10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.236+0000 7f96a61d1700 1 --2- 172.21.15.40:0/3370129519 >> v2:172.21.15.40:3300/0 conn(0x7f96a006ad50 0x7f96a006b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.236+0000 7f96a61d1700 1 -- 172.21.15.40:0/3370129519 >> 172.21.15.40:0/3370129519 conn(0x7f96a00b4aa0 msgr2=0x7f96a00b4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:57.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.236+0000 7f96a61d1700 1 -- 172.21.15.40:0/3370129519 shutdown_connections 2023-09-30T00:47:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.236+0000 7f96a61d1700 1 -- 172.21.15.40:0/3370129519 wait complete. 2023-09-30T00:47:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.237+0000 7f96a61d1700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.237+0000 7f96a61d1700 10 monclient: monmap: 2023-09-30T00:47:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:57.244 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:57.245 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.238+0000 7f96a61d1700 1 librados: starting msgr at 2023-09-30T00:47:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.238+0000 7f96a61d1700 1 librados: starting objecter 2023-09-30T00:47:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.238+0000 7f96a61d1700 1 Processor -- start 2023-09-30T00:47:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.238+0000 7f96a61d1700 1 -- start start 2023-09-30T00:47:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.238+0000 7f96a61d1700 1 librados: setting wanted keys 2023-09-30T00:47:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.238+0000 7f96a61d1700 1 librados: calling monclient init 2023-09-30T00:47:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.238+0000 7f96a61d1700 10 monclient: init 2023-09-30T00:47:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:47:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96a006ad50 0x7f96a00b8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 10 monclient(hunting): picked mon.noname-a con 0x7f96a006ad50 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96a006beb0 0x7f96a0148160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 10 monclient(hunting): picked mon.noname-c con 0x7f96a006beb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96a01432a0 0x7f96a00ba640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 10 monclient(hunting): picked mon.noname-b con 0x7f96a01432a0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a00b8900 con 0x7f96a006ad50 2023-09-30T00:47:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a00b9c60 con 0x7f96a01432a0 2023-09-30T00:47:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a00b9f40 con 0x7f96a006beb0 2023-09-30T00:47:57.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.239+0000 7f96a61d1700 10 monclient(hunting): authenticate will time out at 5338.540764s 2023-09-30T00:47:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.242+0000 7f969effd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96a006beb0 0x7f96a0148160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.243+0000 7f969effd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96a006beb0 0x7f96a0148160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36582/0 (socket says 172.21.15.40:36582) 2023-09-30T00:47:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.243+0000 7f969effd700 1 -- 172.21.15.40:0/4290983236 learned_addr learned my addr 172.21.15.40:0/4290983236 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.243+0000 7f969effd700 10 monclient(hunting): get_auth_request con 0x7f96a006beb0 auth_method 0 2023-09-30T00:47:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.243+0000 7f969effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.243+0000 7f969effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.243+0000 7f969effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.244+0000 7f969effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.244+0000 7f969effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.244+0000 7f969effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.249 INFO:teuthology.orchestra.run.smithi040.stdout:81604378630 2023-09-30T00:47:57.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:47:57.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.250+0000 7f969effd700 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2023-09-30T00:47:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.250+0000 7f969effd700 1 -- 172.21.15.40:0/4290983236 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96a01432a0 msgr2=0x7f96a00ba640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.250+0000 7f969effd700 1 --2- 172.21.15.40:0/4290983236 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96a01432a0 0x7f96a00ba640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.250+0000 7f969effd700 1 -- 172.21.15.40:0/4290983236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96a006ad50 msgr2=0x7f96a00b8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.250+0000 7f969effd700 1 --2- 172.21.15.40:0/4290983236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96a006ad50 0x7f96a00b8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.250+0000 7f969effd700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.250+0000 7f969effd700 1 monclient: found mon.noname-c 2023-09-30T00:47:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.250+0000 7f969effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.250+0000 7f969effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.250+0000 7f969effd700 1 -- 172.21.15.40:0/4290983236 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9698009ae0 con 0x7f96a006beb0 2023-09-30T00:47:57.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.255+0000 7f969effd700 1 --2- 172.21.15.40:0/4290983236 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96a006beb0 0x7f96a0148160 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f969400b7e0 tx=0x7f96940091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.258+0000 7f96877fe700 10 client.?.objecter ms_handle_connect 0x7f96a006beb0 2023-09-30T00:47:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.258+0000 7f96877fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:57.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.263+0000 7f63f3635700 10 client.4327.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T00:47:57.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.263+0000 7f63f3635700 15 client.4327.objecter _session_command_op_assign -1 2 2023-09-30T00:47:57.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.263+0000 7f63f3635700 20 client.4327.objecter _get_session s=0x7f63d006e220 osd=6 3 2023-09-30T00:47:57.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.263+0000 7f63f3635700 20 client.4327.objecter put_session s=0x7f63d006e220 osd=6 3 2023-09-30T00:47:57.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.263+0000 7f63f3635700 20 client.4327.objecter _get_session s=0x7f63d006e220 osd=6 3 2023-09-30T00:47:57.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.263+0000 7f63f3635700 15 client.4327.objecter _session_command_op_remove -1 2 2023-09-30T00:47:57.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.263+0000 7f63f3635700 20 client.4327.objecter get_session s=0x7f63d006e220 osd=6 3 2023-09-30T00:47:57.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.263+0000 7f63f3635700 15 client.4327.objecter _session_command_op_assign 6 2 2023-09-30T00:47:57.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.263+0000 7f63f3635700 20 client.4327.objecter put_session s=0x7f63d006e220 osd=6 4 2023-09-30T00:47:57.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.263+0000 7f63f3635700 10 client.4327.objecter _send_command 2 2023-09-30T00:47:57.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.263+0000 7f63f3635700 1 -- 172.21.15.40:0/1526487877 --> v2:172.21.15.135:6813/117133 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f63ec0adaa0 con 0x7f63d006e8d0 2023-09-30T00:47:57.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96877fe700 1 -- 172.21.15.40:0/4290983236 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969400f510 con 0x7f96a006beb0 2023-09-30T00:47:57.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96877fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:57.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96877fe700 10 monclient: dump: 2023-09-30T00:47:57.267 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.268 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.268 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.268 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.268 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.268 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.268 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.269 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.269 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.269 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.269 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96877fe700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.269 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96877fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.269 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96877fe700 1 -- 172.21.15.40:0/4290983236 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969400f670 con 0x7f96a006beb0 2023-09-30T00:47:57.269 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96877fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.270 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96877fe700 1 -- 172.21.15.40:0/4290983236 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969400fa10 con 0x7f96a006beb0 2023-09-30T00:47:57.270 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96877fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.270 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96877fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:57.270 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96877fe700 10 monclient: dump: 2023-09-30T00:47:57.270 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.270 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.272 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.272 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f969d7fa700 4 set_mon_vals no callback set 2023-09-30T00:47:57.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96a61d1700 5 monclient: authenticate success, global_id 4274 2023-09-30T00:47:57.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96a61d1700 10 monclient: _renew_subs 2023-09-30T00:47:57.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96a61d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.273 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96a61d1700 1 -- 172.21.15.40:0/4290983236 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a00b6f10 con 0x7f96a006beb0 2023-09-30T00:47:57.273 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96a61d1700 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.273 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96a61d1700 10 monclient: _renew_subs 2023-09-30T00:47:57.273 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96a61d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.273 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96a61d1700 1 -- 172.21.15.40:0/4290983236 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a00bab80 con 0x7f96a006beb0 2023-09-30T00:47:57.273 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.264+0000 7f96a61d1700 1 librados: init done 2023-09-30T00:47:57.274 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.267+0000 7f96a61d1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.274 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.267+0000 7f96a61d1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.274 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.267+0000 7f96a61d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.274 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.267+0000 7f96a61d1700 1 -- 172.21.15.40:0/4290983236 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96a0064a00 con 0x7f96a006beb0 2023-09-30T00:47:57.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.271+0000 7f63e17fa700 1 -- 172.21.15.40:0/1526487877 <== osd.6 v2:172.21.15.135:6813/117133 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f63ec0adaa0 con 0x7f63d006e8d0 2023-09-30T00:47:57.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.271+0000 7f63e17fa700 10 client.4327.objecter ms_dispatch 0x7f63ec001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:47:57.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.271+0000 7f63e17fa700 10 client.4327.objecter _finish_command 2 = system:0 2023-09-30T00:47:57.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.271+0000 7f63e17fa700 20 client.4327.objecter put_session s=0x7f63d006e220 osd=6 4 2023-09-30T00:47:57.278 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.271+0000 7f63e17fa700 15 client.4327.objecter _session_command_op_remove 6 2 2023-09-30T00:47:57.278 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 10 librados: watch_flush enter 2023-09-30T00:47:57.278 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 10 librados: watch_flush exit 2023-09-30T00:47:57.278 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 10 client.4327.objecter close_session for osd.6 2023-09-30T00:47:57.278 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 1 -- 172.21.15.40:0/1526487877 >> v2:172.21.15.135:6813/117133 conn(0x7f63d006e8d0 msgr2=0x7f63d0070ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.278 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 1 --2- 172.21.15.40:0/1526487877 >> v2:172.21.15.135:6813/117133 conn(0x7f63d006e8d0 0x7f63d0070ce0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 20 client.4327.objecter put_session s=0x7f63d006e220 osd=6 1 2023-09-30T00:47:57.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 20 client.4327.objecter shutdown clearing up homeless session... 2023-09-30T00:47:57.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 10 client.4327.objecter successfully canceled tick 2023-09-30T00:47:57.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 1 -- 172.21.15.40:0/1526487877 >> v2:172.21.15.135:6800/116426 conn(0x7f63d0068960 msgr2=0x7f63d006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 1 --2- 172.21.15.40:0/1526487877 >> v2:172.21.15.135:6800/116426 conn(0x7f63d0068960 0x7f63d006ae10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f63e80097c0 tx=0x7f63e8006ce0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 10 monclient: shutdown 2023-09-30T00:47:57.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:57.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 1 -- 172.21.15.40:0/1526487877 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63ec13eb60 msgr2=0x7f63ec0ae660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 1 --2- 172.21.15.40:0/1526487877 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f63ec13eb60 0x7f63ec0ae660 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f63dc0094c0 tx=0x7f63dc016000 comp rx=0 tx=0).stop 2023-09-30T00:47:57.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f1bd2700 1 -- 172.21.15.40:0/1526487877 reap_dead start 2023-09-30T00:47:57.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 1 -- 172.21.15.40:0/1526487877 shutdown_connections 2023-09-30T00:47:57.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.274+0000 7f63f3635700 1 -- 172.21.15.40:0/1526487877 >> 172.21.15.40:0/1526487877 conn(0x7f63ec069080 msgr2=0x7f63ec138cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:57.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.276+0000 7f63f3635700 1 -- 172.21.15.40:0/1526487877 shutdown_connections 2023-09-30T00:47:57.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.276+0000 7f63f3635700 1 -- 172.21.15.40:0/1526487877 wait complete. 2023-09-30T00:47:57.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.276+0000 7f63f3635700 1 librados: shutdown 2023-09-30T00:47:57.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.277+0000 7f980effd700 1 -- 172.21.15.40:0/1018353619 <== osd.5 v2:172.21.15.135:6801/117130 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f9814071280 con 0x7f981406e800 2023-09-30T00:47:57.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.277+0000 7f980effd700 10 client.4336.objecter ms_dispatch 0x7f9828001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:47:57.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.277+0000 7f980effd700 10 client.4336.objecter _finish_command 1 = system:0 2023-09-30T00:47:57.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.277+0000 7f980effd700 20 client.4336.objecter put_session s=0x7f981406e150 osd=5 4 2023-09-30T00:47:57.282 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.277+0000 7f980effd700 15 client.4336.objecter _session_command_op_remove 5 1 2023-09-30T00:47:57.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.314+0000 7f982d08d700 10 client.4336.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T00:47:57.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.314+0000 7f982d08d700 15 client.4336.objecter _session_command_op_assign -1 2 2023-09-30T00:47:57.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.314+0000 7f982d08d700 20 client.4336.objecter _get_session s=0x7f981406e150 osd=5 3 2023-09-30T00:47:57.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.314+0000 7f982d08d700 20 client.4336.objecter put_session s=0x7f981406e150 osd=5 3 2023-09-30T00:47:57.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.314+0000 7f982d08d700 20 client.4336.objecter _get_session s=0x7f981406e150 osd=5 3 2023-09-30T00:47:57.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.314+0000 7f982d08d700 15 client.4336.objecter _session_command_op_remove -1 2 2023-09-30T00:47:57.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.314+0000 7f982d08d700 20 client.4336.objecter get_session s=0x7f981406e150 osd=5 3 2023-09-30T00:47:57.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.314+0000 7f982d08d700 15 client.4336.objecter _session_command_op_assign 5 2 2023-09-30T00:47:57.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.314+0000 7f982d08d700 20 client.4336.objecter put_session s=0x7f981406e150 osd=5 4 2023-09-30T00:47:57.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.314+0000 7f982d08d700 10 client.4336.objecter _send_command 2 2023-09-30T00:47:57.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.314+0000 7f982d08d700 1 -- 172.21.15.40:0/1018353619 --> v2:172.21.15.135:6801/117130 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9828146280 con 0x7f981406e800 2023-09-30T00:47:57.317 INFO:teuthology.orchestra.run.smithi040.stdout:60129542151 2023-09-30T00:47:57.318 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T00:47:57.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.322+0000 7f980effd700 1 -- 172.21.15.40:0/1018353619 <== osd.5 v2:172.21.15.135:6801/117130 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9828146280 con 0x7f981406e800 2023-09-30T00:47:57.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.322+0000 7f980effd700 10 client.4336.objecter ms_dispatch 0x7f9828001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:47:57.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.322+0000 7f980effd700 10 client.4336.objecter _finish_command 2 = system:0 2023-09-30T00:47:57.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.322+0000 7f980effd700 20 client.4336.objecter put_session s=0x7f981406e150 osd=5 4 2023-09-30T00:47:57.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.322+0000 7f980effd700 15 client.4336.objecter _session_command_op_remove 5 2 2023-09-30T00:47:57.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.328+0000 7f982d08d700 10 librados: watch_flush enter 2023-09-30T00:47:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.328+0000 7f982d08d700 10 librados: watch_flush exit 2023-09-30T00:47:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.328+0000 7f982d08d700 10 client.4336.objecter close_session for osd.5 2023-09-30T00:47:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.328+0000 7f982d08d700 1 -- 172.21.15.40:0/1018353619 >> v2:172.21.15.135:6801/117130 conn(0x7f981406e800 msgr2=0x7f9814070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.328+0000 7f982d08d700 1 --2- 172.21.15.40:0/1018353619 >> v2:172.21.15.135:6801/117130 conn(0x7f981406e800 0x7f9814070c10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.342 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.328+0000 7f982d08d700 20 client.4336.objecter put_session s=0x7f981406e150 osd=5 1 2023-09-30T00:47:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.328+0000 7f982d08d700 20 client.4336.objecter shutdown clearing up homeless session... 2023-09-30T00:47:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.328+0000 7f982d08d700 10 client.4336.objecter successfully canceled tick 2023-09-30T00:47:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.335+0000 7f982d08d700 1 -- 172.21.15.40:0/1018353619 >> v2:172.21.15.135:6800/116426 conn(0x7f9814068890 msgr2=0x7f981406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.335+0000 7f982d08d700 1 --2- 172.21.15.40:0/1018353619 >> v2:172.21.15.135:6800/116426 conn(0x7f9814068890 0x7f981406ad40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f981c006fd0 tx=0x7f981c006db0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.335+0000 7f982d08d700 10 monclient: shutdown 2023-09-30T00:47:57.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.335+0000 7f982d08d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.335+0000 7f982d08d700 1 -- 172.21.15.40:0/1018353619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98280b1870 msgr2=0x7f9828155ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.335+0000 7f982d08d700 1 --2- 172.21.15.40:0/1018353619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f98280b1870 0x7f9828155ae0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f982806b450 tx=0x7f9820018000 comp rx=0 tx=0).stop 2023-09-30T00:47:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.338+0000 7f982759e700 1 -- 172.21.15.40:0/1018353619 reap_dead start 2023-09-30T00:47:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.338+0000 7f982d08d700 1 -- 172.21.15.40:0/1018353619 shutdown_connections 2023-09-30T00:47:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.338+0000 7f982d08d700 1 -- 172.21.15.40:0/1018353619 >> 172.21.15.40:0/1018353619 conn(0x7f98280631f0 msgr2=0x7f9828144de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:57.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.340+0000 7f982d08d700 1 -- 172.21.15.40:0/1018353619 shutdown_connections 2023-09-30T00:47:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.341+0000 7f982d08d700 1 -- 172.21.15.40:0/1018353619 wait complete. 2023-09-30T00:47:57.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.341+0000 7f982d08d700 1 librados: shutdown 2023-09-30T00:47:57.363 INFO:teuthology.orchestra.run.smithi040.stdout:64424509447 2023-09-30T00:47:57.364 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T00:47:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.380+0000 7fa9a3335700 10 monclient: get_monmap_and_config 2023-09-30T00:47:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.380+0000 7fa9a3335700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.380+0000 7fa9a3335700 10 monclient: monmap: 2023-09-30T00:47:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:57.381231+0000 2023-09-30T00:47:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:57.381231+0000 2023-09-30T00:47:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:57.398 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.382+0000 7fa9a3335700 1 Processor -- start 2023-09-30T00:47:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.382+0000 7fa9a3335700 1 -- start start 2023-09-30T00:47:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.382+0000 7fa9a3335700 10 monclient: init 2023-09-30T00:47:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.383+0000 7fa9a3335700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.383+0000 7fa9a3335700 10 monclient: _add_conns ranks=[4,1,0,3,2] 2023-09-30T00:47:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.383+0000 7fa9a3335700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa99c0b0f50 0x7fa99c06e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.384+0000 7fa9a3335700 10 monclient(hunting): picked mon.noname-c con 0x7fa99c0b0f50 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.384+0000 7fa9a3335700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa99c06f840 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.384+0000 7fa9a3335700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa99c06e980 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.384+0000 7fa9a3335700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c0abe10 con 0x7fa99c06e980 2023-09-30T00:47:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.384+0000 7fa9a3335700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c06d4d0 con 0x7fa99c06f840 2023-09-30T00:47:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.384+0000 7fa9a3335700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.384+0000 7fa9a3335700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99c06d7b0 con 0x7fa99c0b0f50 2023-09-30T00:47:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.384+0000 7fa9a3335700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.384+0000 7fa9a3335700 10 monclient(hunting): authenticate will time out at 5338.685095s 2023-09-30T00:47:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.385+0000 7fa9a10d1700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa99c0b0f50 0x7fa99c06e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.385+0000 7fa9a10d1700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa99c0b0f50 0x7fa99c06e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43270/0 (socket says 172.21.15.40:43270) 2023-09-30T00:47:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.385+0000 7fa9a10d1700 1 -- 172.21.15.40:0/3855286636 learned_addr learned my addr 172.21.15.40:0/3855286636 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.385+0000 7fa9a10d1700 10 monclient(hunting): get_auth_request con 0x7fa99c0b0f50 auth_method 0 2023-09-30T00:47:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.385+0000 7fa9a10d1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.385+0000 7fa9a10d1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.385+0000 7fa9a10d1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.386+0000 7fa9a10d1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.389+0000 7fa993fff700 1 -- 172.21.15.40:0/3855286636 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896811668 0 0) 0x7fa99c06d4d0 con 0x7fa99c06f840 2023-09-30T00:47:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.386+0000 7fa9a10d1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.395+0000 7fa9a10d1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.395+0000 7fa993fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.395+0000 7fa993fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.395+0000 7fa993fff700 10 monclient(hunting): my global_id is 4342 2023-09-30T00:47:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.395+0000 7fa993fff700 1 -- 172.21.15.40:0/3855286636 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa988003550 con 0x7fa99c06f840 2023-09-30T00:47:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.401+0000 7fa9a10d1700 10 monclient(hunting): handle_auth_done global_id 4339 payload 274 2023-09-30T00:47:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.401+0000 7fa9a10d1700 1 -- 172.21.15.40:0/3855286636 >> v1:172.21.15.135:6789/0 conn(0x7fa99c06f840 legacy=0x7fa99c0aaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.401+0000 7fa9a10d1700 1 -- 172.21.15.40:0/3855286636 >> v1:172.21.15.40:6789/0 conn(0x7fa99c06e980 legacy=0x7fa99c0ab700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.401+0000 7fa9a10d1700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.401+0000 7fa9a10d1700 1 monclient: found mon.noname-c 2023-09-30T00:47:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.401+0000 7fa9a10d1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.401+0000 7fa9a10d1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.401+0000 7fa9a10d1700 1 -- 172.21.15.40:0/3855286636 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c0aac80 con 0x7fa99c0b0f50 2023-09-30T00:47:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.403+0000 7fa9a10d1700 1 --2- 172.21.15.40:0/3855286636 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c0b0f50 0x7fa99c06e440 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa994009b90 tx=0x7fa994009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8794853cb11ecf9a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa993fff700 1 -- 172.21.15.40:0/3855286636 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9940092e0 con 0x7fa99c0b0f50 2023-09-30T00:47:57.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa993fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa993fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:57.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa993fff700 10 monclient: dump: 2023-09-30T00:47:57.410 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.410 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.410 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.410 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.410 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.410 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.410 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.411 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.411 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.411 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa993fff700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa993fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa993fff700 1 -- 172.21.15.40:0/3855286636 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99400c9d0 con 0x7fa99c0b0f50 2023-09-30T00:47:57.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa993fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa993fff700 1 -- 172.21.15.40:0/3855286636 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa99400cd90 con 0x7fa99c0b0f50 2023-09-30T00:47:57.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa993fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa993fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:57.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa993fff700 10 monclient: dump: 2023-09-30T00:47:57.412 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.412 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.412 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.412 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.412 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.412 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.412 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.412 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.412 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.412 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa9a3335700 5 monclient: authenticate success, global_id 4339 2023-09-30T00:47:57.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa9a3335700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:57.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.409+0000 7fa9a3335700 4 set_mon_vals no callback set 2023-09-30T00:47:57.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.410+0000 7fa9a3335700 10 monclient: shutdown 2023-09-30T00:47:57.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.410+0000 7fa9a3335700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:57.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.410+0000 7fa9a3335700 1 -- 172.21.15.40:0/3855286636 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c0b0f50 msgr2=0x7fa99c06e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.410+0000 7fa9a3335700 1 --2- 172.21.15.40:0/3855286636 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c0b0f50 0x7fa99c06e440 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa994009b90 tx=0x7fa994009f90 comp rx=0 tx=0).stop 2023-09-30T00:47:57.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.410+0000 7fa9a3335700 1 -- 172.21.15.40:0/3855286636 shutdown_connections 2023-09-30T00:47:57.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.410+0000 7fa9a3335700 1 --2- 172.21.15.40:0/3855286636 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c0b0f50 0x7fa99c06e440 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.410+0000 7fa9a3335700 1 -- 172.21.15.40:0/3855286636 >> 172.21.15.40:0/3855286636 conn(0x7fa99c067bf0 msgr2=0x7fa99c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:57.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.410+0000 7fa9a3335700 1 -- 172.21.15.40:0/3855286636 shutdown_connections 2023-09-30T00:47:57.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.410+0000 7fa9a3335700 1 -- 172.21.15.40:0/3855286636 wait complete. 2023-09-30T00:47:57.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.411+0000 7fa9a3335700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.411+0000 7fa9a3335700 10 monclient: monmap: 2023-09-30T00:47:57.415 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.415 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:57.415 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:57.415 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:57.415 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.415 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:57.415 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:57.416 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:57.416 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:57.416 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.411+0000 7fa9a3335700 1 librados: starting msgr at 2023-09-30T00:47:57.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.411+0000 7fa9a3335700 1 librados: starting objecter 2023-09-30T00:47:57.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.411+0000 7fa9a3335700 1 Processor -- start 2023-09-30T00:47:57.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.412+0000 7fa9a3335700 1 -- start start 2023-09-30T00:47:57.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.412+0000 7fa9a3335700 1 librados: setting wanted keys 2023-09-30T00:47:57.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.412+0000 7fa9a3335700 1 librados: calling monclient init 2023-09-30T00:47:57.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.412+0000 7fa9a3335700 10 monclient: init 2023-09-30T00:47:57.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:47:57.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa99c06e980 0x7fa99c077f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 10 monclient(hunting): picked mon.noname-a con 0x7fa99c06e980 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:57.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa99c06f840 0x7fa99c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 10 monclient(hunting): picked mon.noname-c con 0x7fa99c06f840 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c0b0f50 0x7fa99c15e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 10 monclient(hunting): picked mon.noname-b con 0x7fa99c0b0f50 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:57.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0794a0 con 0x7fa99c06e980 2023-09-30T00:47:57.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c0a8120 con 0x7fa99c0b0f50 2023-09-30T00:47:57.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c07a160 con 0x7fa99c06f840 2023-09-30T00:47:57.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.413+0000 7fa9a3335700 10 monclient(hunting): authenticate will time out at 5338.714123s 2023-09-30T00:47:57.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.414+0000 7fa9a18d2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c0b0f50 0x7fa99c15e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.414+0000 7fa9a18d2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c0b0f50 0x7fa99c15e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43286/0 (socket says 172.21.15.40:43286) 2023-09-30T00:47:57.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.414+0000 7fa9a18d2700 1 -- 172.21.15.40:0/891003756 learned_addr learned my addr 172.21.15.40:0/891003756 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.414+0000 7fa9a18d2700 10 monclient(hunting): get_auth_request con 0x7fa99c0b0f50 auth_method 0 2023-09-30T00:47:57.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.414+0000 7fa9a18d2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.414+0000 7fa9a18d2700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.414+0000 7fa9a18d2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.415+0000 7fa9a18d2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.415+0000 7fa9a18d2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.415+0000 7fa9a18d2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.419+0000 7fa9a18d2700 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2023-09-30T00:47:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.419+0000 7fa9a18d2700 1 -- 172.21.15.40:0/891003756 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa99c06f840 msgr2=0x7fa99c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.419+0000 7fa9a18d2700 1 --2- 172.21.15.40:0/891003756 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa99c06f840 0x7fa99c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.419+0000 7fa9a18d2700 1 -- 172.21.15.40:0/891003756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa99c06e980 msgr2=0x7fa99c077f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.419+0000 7fa9a18d2700 1 --2- 172.21.15.40:0/891003756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa99c06e980 0x7fa99c077f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.419+0000 7fa9a18d2700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.419+0000 7fa9a18d2700 1 monclient: found mon.noname-b 2023-09-30T00:47:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.419+0000 7fa9a18d2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.419+0000 7fa9a18d2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.419+0000 7fa9a18d2700 1 -- 172.21.15.40:0/891003756 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9940096d0 con 0x7fa99c0b0f50 2023-09-30T00:47:57.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.421+0000 7fa9a18d2700 1 --2- 172.21.15.40:0/891003756 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c0b0f50 0x7fa99c15e970 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00b5e0 tx=0x7fa98c0090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.421+0000 7fa9917fa700 10 client.?.objecter ms_handle_connect 0x7fa99c0b0f50 2023-09-30T00:47:57.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.421+0000 7fa9917fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:57.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.425+0000 7f96877fe700 1 -- 172.21.15.40:0/4290983236 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9694024430 con 0x7f96a006beb0 2023-09-30T00:47:57.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.426+0000 7f96877fe700 1 --2- 172.21.15.40:0/4290983236 >> v2:172.21.15.135:6800/116426 conn(0x7f968c0688c0 0x7f968c06ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.426+0000 7f96877fe700 1 -- 172.21.15.40:0/4290983236 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f96940844d0 con 0x7f96a006beb0 2023-09-30T00:47:57.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.426+0000 7f96877fe700 10 client.4274.objecter ms_dispatch 0x7f96a0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:57.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.426+0000 7f96877fe700 3 client.4274.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:57.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.426+0000 7f96877fe700 3 client.4274.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:57.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.426+0000 7f96877fe700 20 client.4274.objecter dump_active .. 0 homeless 2023-09-30T00:47:57.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.426+0000 7fa9917fa700 1 -- 172.21.15.40:0/891003756 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c00f510 con 0x7fa99c0b0f50 2023-09-30T00:47:57.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.426+0000 7fa9917fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.426+0000 7fa9917fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:57.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.426+0000 7fa9917fa700 10 monclient: dump: 2023-09-30T00:47:57.430 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.431 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.431 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.431 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.431 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.431 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.431 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.431 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.431 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.431 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.426+0000 7fa9917fa700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.426+0000 7fa9917fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.426+0000 7fa9917fa700 1 -- 172.21.15.40:0/891003756 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98c00f690 con 0x7fa99c0b0f50 2023-09-30T00:47:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.427+0000 7fa9917fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.427+0000 7fa9917fa700 1 -- 172.21.15.40:0/891003756 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c00fa50 con 0x7fa99c0b0f50 2023-09-30T00:47:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.427+0000 7fa9917fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.427+0000 7fa9917fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.427+0000 7fa9917fa700 10 monclient: dump: 2023-09-30T00:47:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.432 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.433 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.433 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.433 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.433 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.433 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.433 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.433 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.427+0000 7fa9a3335700 5 monclient: authenticate success, global_id 4345 2023-09-30T00:47:57.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.427+0000 7fa9a3335700 10 monclient: _renew_subs 2023-09-30T00:47:57.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.427+0000 7fa9a3335700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.427+0000 7fa9a3335700 1 -- 172.21.15.40:0/891003756 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c07a3d0 con 0x7fa99c0b0f50 2023-09-30T00:47:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.427+0000 7fa9a3335700 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.427+0000 7fa9a3335700 10 monclient: _renew_subs 2023-09-30T00:47:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.427+0000 7fa9a3335700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.427+0000 7fa9a3335700 1 -- 172.21.15.40:0/891003756 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c077130 con 0x7fa99c0b0f50 2023-09-30T00:47:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.427+0000 7fa9a3335700 1 librados: init done 2023-09-30T00:47:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.428+0000 7fa9a3335700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.428+0000 7fa9a3335700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.428+0000 7fa9a3335700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.428+0000 7fa9a3335700 1 -- 172.21.15.40:0/891003756 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99c072b90 con 0x7fa99c0b0f50 2023-09-30T00:47:57.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.429+0000 7f969f7fe700 1 --2- 172.21.15.40:0/4290983236 >> v2:172.21.15.135:6800/116426 conn(0x7f968c0688c0 0x7f968c06ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.429+0000 7f969f7fe700 10 monclient: get_auth_request con 0x7f968c0688c0 auth_method 0 2023-09-30T00:47:57.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.431+0000 7f969f7fe700 1 --2- 172.21.15.40:0/4290983236 >> v2:172.21.15.135:6800/116426 conn(0x7f968c0688c0 0x7f968c06ad70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9690006fd0 tx=0x7f9690006db0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.431+0000 7f96877fe700 10 client.4274.objecter ms_handle_connect 0x7f968c0688c0 2023-09-30T00:47:57.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.431+0000 7f1b35ffb700 1 -- 172.21.15.40:0/846249853 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1b3001ca80 con 0x7f1b400a7610 2023-09-30T00:47:57.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.431+0000 7f1b35ffb700 1 --2- 172.21.15.40:0/846249853 >> v2:172.21.15.135:6800/116426 conn(0x7f1b240689e0 0x7f1b2406ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.431+0000 7f1b35ffb700 1 -- 172.21.15.40:0/846249853 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1b30086220 con 0x7f1b400a7610 2023-09-30T00:47:57.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.431+0000 7f1b35ffb700 10 client.4302.objecter ms_dispatch 0x7f1b40001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:57.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.431+0000 7f1b35ffb700 3 client.4302.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:57.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.431+0000 7f1b35ffb700 3 client.4302.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:57.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.431+0000 7f1b35ffb700 20 client.4302.objecter dump_active .. 0 homeless 2023-09-30T00:47:57.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.431+0000 7fa992ffd700 4 set_mon_vals no callback set 2023-09-30T00:47:57.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.433+0000 7f1b4575c700 1 --2- 172.21.15.40:0/846249853 >> v2:172.21.15.135:6800/116426 conn(0x7f1b240689e0 0x7f1b2406ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.433+0000 7f1b4575c700 10 monclient: get_auth_request con 0x7f1b240689e0 auth_method 0 2023-09-30T00:47:57.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.441+0000 7f1b4575c700 1 --2- 172.21.15.40:0/846249853 >> v2:172.21.15.135:6800/116426 conn(0x7f1b240689e0 0x7f1b2406ae90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c006fd0 tx=0x7f1b3c006db0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.443+0000 7f1b35ffb700 10 client.4302.objecter ms_handle_connect 0x7f1b240689e0 2023-09-30T00:47:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.449+0000 7f8333ad7700 10 monclient: get_monmap_and_config 2023-09-30T00:47:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.449+0000 7f8333ad7700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.449+0000 7f8333ad7700 10 monclient: monmap: 2023-09-30T00:47:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.451 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:57.450582+0000 2023-09-30T00:47:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:57.450582+0000 2023-09-30T00:47:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:57.452 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:57.452 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.460+0000 7f8333ad7700 1 Processor -- start 2023-09-30T00:47:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.460+0000 7f8333ad7700 1 -- start start 2023-09-30T00:47:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.460+0000 7f8333ad7700 10 monclient: init 2023-09-30T00:47:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.461+0000 7f8333ad7700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.461+0000 7f8333ad7700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-09-30T00:47:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.461+0000 7f8333ad7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f832c0b0f50 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.461+0000 7f8333ad7700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f832c06f0f0 0x7f832c06e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.461+0000 7f8333ad7700 10 monclient(hunting): picked mon.noname-c con 0x7f832c06f0f0 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.461+0000 7f8333ad7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f832c145d30 0x7f832c1461b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.461+0000 7f8333ad7700 10 monclient(hunting): picked mon.noname-a con 0x7f832c145d30 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.461+0000 7f8333ad7700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832c06e970 con 0x7f832c0b0f50 2023-09-30T00:47:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.461+0000 7f8333ad7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.461+0000 7f8333ad7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f832c06d760 con 0x7f832c145d30 2023-09-30T00:47:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.461+0000 7f8333ad7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.461+0000 7f8333ad7700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f832c06ebd0 con 0x7f832c06f0f0 2023-09-30T00:47:57.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.461+0000 7f8333ad7700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.461+0000 7f8333ad7700 10 monclient(hunting): authenticate will time out at 5338.762825s 2023-09-30T00:47:57.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.482+0000 7f8ca65cb700 10 monclient: get_monmap_and_config 2023-09-30T00:47:57.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.482+0000 7f8ca65cb700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.482+0000 7f8ca65cb700 10 monclient: monmap: 2023-09-30T00:47:57.483 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.483 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.483 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:57.483501+0000 2023-09-30T00:47:57.484 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:57.483501+0000 2023-09-30T00:47:57.484 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.484 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.484 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:57.484 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:57.484 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:57.484 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:57.484 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:57.484 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.490+0000 7f8ca65cb700 1 Processor -- start 2023-09-30T00:47:57.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.490+0000 7f8ca65cb700 1 -- start start 2023-09-30T00:47:57.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.490+0000 7f8ca65cb700 10 monclient: init 2023-09-30T00:47:57.492 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.491+0000 7f8ca65cb700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.492 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.491+0000 7f8ca65cb700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-09-30T00:47:57.492 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.491+0000 7f8ca65cb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ca00b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:57.492 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.491+0000 7f8ca65cb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ca00b1830 0x7f8ca00a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.491+0000 7f8ca65cb700 10 monclient(hunting): picked mon.noname-b con 0x7f8ca00b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.491+0000 7f8ca65cb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ca013ac20 0x7f8ca013b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.491+0000 7f8ca65cb700 10 monclient(hunting): picked mon.noname-a con 0x7f8ca013ac20 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.491+0000 7f8ca65cb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca00a5b20 con 0x7f8ca00b1460 2023-09-30T00:47:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.491+0000 7f8ca65cb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.491+0000 7f8ca65cb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca0141e40 con 0x7f8ca013ac20 2023-09-30T00:47:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.491+0000 7f8ca65cb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.491+0000 7f8ca65cb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca00a64f0 con 0x7f8ca00b1830 2023-09-30T00:47:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.491+0000 7f8ca65cb700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.491+0000 7f8ca65cb700 10 monclient(hunting): authenticate will time out at 5338.792523s 2023-09-30T00:47:57.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.503+0000 7f7512867700 10 monclient: get_monmap_and_config 2023-09-30T00:47:57.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.503+0000 7f7512867700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.503+0000 7f7512867700 10 monclient: monmap: 2023-09-30T00:47:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:57.504265+0000 2023-09-30T00:47:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:57.504265+0000 2023-09-30T00:47:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:57.505 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:57.505 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.504+0000 7f7512867700 1 Processor -- start 2023-09-30T00:47:57.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.504+0000 7f7512867700 1 -- start start 2023-09-30T00:47:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.504+0000 7f7512867700 10 monclient: init 2023-09-30T00:47:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.505+0000 7f7512867700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.505+0000 7f7512867700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-09-30T00:47:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.505+0000 7f7512867700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f750c0b14a0 0x7f750c0afe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.505+0000 7f7512867700 10 monclient(hunting): picked mon.noname-b con 0x7f750c0b14a0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.505+0000 7f7512867700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f750c0b1870 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.505+0000 7f7512867700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f750c06dd20 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.505+0000 7f7512867700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750c13ff40 con 0x7f750c0b1870 2023-09-30T00:47:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.505+0000 7f7512867700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750c06b030 con 0x7f750c06dd20 2023-09-30T00:47:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.505+0000 7f7512867700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.505+0000 7f7512867700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750c06b310 con 0x7f750c0b14a0 2023-09-30T00:47:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.505+0000 7f7512867700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.505+0000 7f7512867700 10 monclient(hunting): authenticate will time out at 5338.806522s 2023-09-30T00:47:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.578+0000 7f8c977fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ca00b1830 0x7f8ca00a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.579+0000 7f8c977fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ca00b1830 0x7f8ca00a5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36600/0 (socket says 172.21.15.40:36600) 2023-09-30T00:47:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.579+0000 7f8c977fe700 1 -- 172.21.15.40:0/1038578848 learned_addr learned my addr 172.21.15.40:0/1038578848 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.579+0000 7f8c977fe700 10 monclient(hunting): get_auth_request con 0x7f8ca00b1830 auth_method 0 2023-09-30T00:47:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.579+0000 7f8c977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.579+0000 7f8c977fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.579+0000 7f8c977fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.579+0000 7f8c977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.579+0000 7f8c977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.579+0000 7f8c977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.579+0000 7f750bfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f750c0b14a0 0x7f750c0afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.584+0000 7fc668f81700 10 monclient: get_monmap_and_config 2023-09-30T00:47:57.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.584+0000 7fc668f81700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.584+0000 7fc668f81700 10 monclient: monmap: 2023-09-30T00:47:57.585 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.585 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.585 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:57.585116+0000 2023-09-30T00:47:57.585 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:57.585116+0000 2023-09-30T00:47:57.585 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.585 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.585 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:57.585 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:57.585 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:57.585 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:57.586 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:57.586 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.584+0000 7fc668f81700 1 Processor -- start 2023-09-30T00:47:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.584+0000 7fc668f81700 1 -- start start 2023-09-30T00:47:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.584+0000 7fc668f81700 10 monclient: init 2023-09-30T00:47:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.585+0000 7fc668f81700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.585+0000 7fc668f81700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-09-30T00:47:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.585+0000 7fc668f81700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640af460 0x7fc66413f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.585+0000 7fc668f81700 10 monclient(hunting): picked mon.noname-b con 0x7fc6640af460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.585+0000 7fc668f81700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc66406ee80 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.585+0000 7fc668f81700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6640abc80 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.585+0000 7fc668f81700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6640ad360 con 0x7fc6640abc80 2023-09-30T00:47:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.585+0000 7fc668f81700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66413fb10 con 0x7fc66406ee80 2023-09-30T00:47:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.585+0000 7fc668f81700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.585+0000 7fc668f81700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6640ae6a0 con 0x7fc6640af460 2023-09-30T00:47:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.585+0000 7fc668f81700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.585+0000 7fc668f81700 10 monclient(hunting): authenticate will time out at 5338.886794s 2023-09-30T00:47:57.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.607+0000 7f750bfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f750c0b14a0 0x7f750c0afe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36616/0 (socket says 172.21.15.40:36616) 2023-09-30T00:47:57.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.607+0000 7f750bfff700 1 -- 172.21.15.40:0/698955877 learned_addr learned my addr 172.21.15.40:0/698955877 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.607+0000 7f750bfff700 10 monclient(hunting): get_auth_request con 0x7f750c0b14a0 auth_method 0 2023-09-30T00:47:57.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.607+0000 7f750bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.607+0000 7f750bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.607+0000 7f750bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.609+0000 7f8c977fe700 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2023-09-30T00:47:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.609+0000 7f8c977fe700 1 -- 172.21.15.40:0/1038578848 >> v2:172.21.15.40:3300/0 conn(0x7f8ca013ac20 msgr2=0x7f8ca013b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.609+0000 7f8c977fe700 1 --2- 172.21.15.40:0/1038578848 >> v2:172.21.15.40:3300/0 conn(0x7f8ca013ac20 0x7f8ca013b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.609+0000 7f8c977fe700 1 -- 172.21.15.40:0/1038578848 >> v1:172.21.15.40:6789/0 conn(0x7f8ca00b1460 legacy=0x7f8ca0141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.609+0000 7f8c977fe700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.609+0000 7f8c977fe700 1 monclient: found mon.noname-b 2023-09-30T00:47:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.609+0000 7f8c977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.609+0000 7f8c977fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.609+0000 7f8c977fe700 1 -- 172.21.15.40:0/1038578848 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca00a71f0 con 0x7f8ca00b1830 2023-09-30T00:47:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.619+0000 7f7d8d530700 10 monclient: get_monmap_and_config 2023-09-30T00:47:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.619+0000 7f7d8d530700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.620+0000 7f7d8d530700 10 monclient: monmap: 2023-09-30T00:47:57.620 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:57.621049+0000 2023-09-30T00:47:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:57.621049+0000 2023-09-30T00:47:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:57.621 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:57.622 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.620+0000 7f7d8d530700 1 Processor -- start 2023-09-30T00:47:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.620+0000 7f7d8d530700 1 -- start start 2023-09-30T00:47:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.620+0000 7f7d8d530700 10 monclient: init 2023-09-30T00:47:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.621+0000 7f7d8d530700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.621+0000 7f7d8d530700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-09-30T00:47:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.621+0000 7f7d8d530700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d880b1460 0x7f7d880b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.621+0000 7f7d8d530700 10 monclient(hunting): picked mon.noname-b con 0x7f7d880b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.621+0000 7f7d8d530700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d880a5fb0 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.621+0000 7f7d8d530700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d8813ac20 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.621+0000 7f7d8d530700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d880a5b20 con 0x7f7d880a5fb0 2023-09-30T00:47:57.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.621+0000 7f7d8d530700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d880a71f0 con 0x7f7d8813ac20 2023-09-30T00:47:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.621+0000 7f7d8d530700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.621+0000 7f7d8d530700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880a74d0 con 0x7f7d880b1460 2023-09-30T00:47:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.621+0000 7f7d8d530700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.621+0000 7f7d8d530700 10 monclient(hunting): authenticate will time out at 5338.922777s 2023-09-30T00:47:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.658+0000 7f750bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.658+0000 7f750bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.658+0000 7f750bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.665+0000 7fc662d9d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc6640abc80 0x7fc6640ac120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43370/0 (socket says 172.21.15.40:43370) 2023-09-30T00:47:57.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.665+0000 7fc662d9d700 1 -- 172.21.15.40:0/317490228 learned_addr learned my addr 172.21.15.40:0/317490228 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.669+0000 7f8332074700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f832c145d30 0x7f832c1461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.669+0000 7f7d867fc700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7d880a5fb0 0x7f7d88141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43384/0 (socket says 172.21.15.40:43384) 2023-09-30T00:47:57.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.669+0000 7f7d867fc700 1 -- 172.21.15.40:0/4218724934 learned_addr learned my addr 172.21.15.40:0/4218724934 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:57.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.675+0000 7f8332074700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f832c145d30 0x7f832c1461b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56602/0 (socket says 172.21.15.40:56602) 2023-09-30T00:47:57.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.675+0000 7f8332074700 1 -- 172.21.15.40:0/2639070419 learned_addr learned my addr 172.21.15.40:0/2639070419 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.675+0000 7f8332074700 10 monclient(hunting): get_auth_request con 0x7f832c145d30 auth_method 0 2023-09-30T00:47:57.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.675+0000 7f8332074700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.675+0000 7f8332074700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.675+0000 7f8332074700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.684+0000 7f8332074700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.684+0000 7f8332074700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.684+0000 7f8332074700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.695+0000 7f8332074700 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2023-09-30T00:47:57.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.695+0000 7f8332074700 1 -- 172.21.15.40:0/2639070419 >> v2:172.21.15.135:3300/0 conn(0x7f832c06f0f0 msgr2=0x7f832c06e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.695+0000 7f8332074700 1 --2- 172.21.15.40:0/2639070419 >> v2:172.21.15.135:3300/0 conn(0x7f832c06f0f0 0x7f832c06e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.695+0000 7f8332074700 1 -- 172.21.15.40:0/2639070419 >> v1:172.21.15.135:6789/0 conn(0x7f832c0b0f50 legacy=0x7f832c06b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.695+0000 7f8332074700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.695+0000 7f8332074700 1 monclient: found mon.noname-a 2023-09-30T00:47:57.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.695+0000 7f8332074700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.695+0000 7f8332074700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:57.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.695+0000 7f8332074700 1 -- 172.21.15.40:0/2639070419 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c0b1320 con 0x7f832c145d30 2023-09-30T00:47:57.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.697+0000 7fc66159a700 1 -- 172.21.15.40:0/317490228 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532747047 0 0) 0x7fc6640ad360 con 0x7fc6640abc80 2023-09-30T00:47:57.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.697+0000 7fc66159a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.697+0000 7fc66159a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.697+0000 7fc66159a700 10 monclient(hunting): my global_id is 4308 2023-09-30T00:47:57.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.697+0000 7fc66159a700 1 -- 172.21.15.40:0/317490228 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc64c003570 con 0x7fc6640abc80 2023-09-30T00:47:57.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.704+0000 7fa9917fa700 1 -- 172.21.15.40:0/891003756 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa98c024430 con 0x7fa99c0b0f50 2023-09-30T00:47:57.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.704+0000 7fa9917fa700 1 --2- 172.21.15.40:0/891003756 >> v2:172.21.15.135:6800/116426 conn(0x7fa980068960 0x7fa98006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.704+0000 7fa9917fa700 1 -- 172.21.15.40:0/891003756 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa98c084450 con 0x7fa99c0b0f50 2023-09-30T00:47:57.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.704+0000 7fa9917fa700 10 client.4345.objecter ms_dispatch 0x7fa99c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:57.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.704+0000 7fa9917fa700 3 client.4345.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:57.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.704+0000 7fa9917fa700 3 client.4345.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:57.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.704+0000 7fa9a10d1700 1 --2- 172.21.15.40:0/891003756 >> v2:172.21.15.135:6800/116426 conn(0x7fa980068960 0x7fa98006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.704+0000 7fa9917fa700 20 client.4345.objecter dump_active .. 0 homeless 2023-09-30T00:47:57.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.704+0000 7fa9a10d1700 10 monclient: get_auth_request con 0x7fa980068960 auth_method 0 2023-09-30T00:47:57.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.705+0000 7fa9a10d1700 1 --2- 172.21.15.40:0/891003756 >> v2:172.21.15.135:6800/116426 conn(0x7fa980068960 0x7fa98006ae10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa994009b90 tx=0x7fa994018000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.705+0000 7fa9917fa700 10 client.4345.objecter ms_handle_connect 0x7fa980068960 2023-09-30T00:47:57.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.712+0000 7f8332074700 1 --2- 172.21.15.40:0/2639070419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f832c145d30 0x7f832c1461b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f83280099b0 tx=0x7f8328009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d996e26b9b7e26e7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fa9917fa700 1 -- 172.21.15.40:0/891003756 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa98c051d70 con 0x7fa99c0b0f50 2023-09-30T00:47:57.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fa9917fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fa9917fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:57.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 -- 172.21.15.40:0/317490228 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3342364371 0 0) 0x7fc64c003570 con 0x7fc6640abc80 2023-09-30T00:47:57.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 -- 172.21.15.40:0/317490228 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640af460 msgr2=0x7fc66413f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 --2- 172.21.15.40:0/317490228 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640af460 0x7fc66413f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 -- 172.21.15.40:0/317490228 >> v1:172.21.15.135:6789/0 conn(0x7fc66406ee80 legacy=0x7fc6640ad7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 monclient: found mon.noname-a-legacy 2023-09-30T00:47:57.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2023-09-30T00:47:57.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 -- 172.21.15.40:0/317490228 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6640ae8a0 con 0x7fc6640abc80 2023-09-30T00:47:57.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 -- 172.21.15.40:0/317490228 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3033000540 0 0) 0x7fc65c0017f0 con 0x7fc6640abc80 2023-09-30T00:47:57.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc668f81700 5 monclient: authenticate success, global_id 4308 2023-09-30T00:47:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc668f81700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T00:47:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient: dump: 2023-09-30T00:47:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.718 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2023-09-30T00:47:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 -- 172.21.15.40:0/317490228 >> v1:172.21.15.40:6789/0 conn(0x7fc6640abc80 legacy=0x7fc6640ac120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:47:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 --2- 172.21.15.40:0/317490228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64c003b10 0x7fc64c003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient(hunting): picked mon.a con 0x7fc64c003b10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 --2- 172.21.15.40:0/317490228 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640af460 0x7fc64c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient(hunting): picked mon.c con 0x7fc6640af460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 --2- 172.21.15.40:0/317490228 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc64c004f50 0x7fc64c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient(hunting): picked mon.b con 0x7fc64c004f50 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:57.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 -- 172.21.15.40:0/317490228 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6640ae6a0 con 0x7fc64c003b10 2023-09-30T00:47:57.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 -- 172.21.15.40:0/317490228 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c005620 con 0x7fc64c004f50 2023-09-30T00:47:57.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 1 -- 172.21.15.40:0/317490228 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c00bfa0 con 0x7fc6640af460 2023-09-30T00:47:57.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc66159a700 10 monclient(hunting): _finish_auth 0 2023-09-30T00:47:57.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7fc668f81700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:57.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.713+0000 7f750bfff700 10 monclient(hunting): handle_auth_done global_id 4280 payload 274 2023-09-30T00:47:57.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.714+0000 7f750bfff700 1 -- 172.21.15.40:0/698955877 >> v1:172.21.15.135:6789/0 conn(0x7f750c06dd20 legacy=0x7f750c14cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.714+0000 7f750bfff700 1 -- 172.21.15.40:0/698955877 >> v1:172.21.15.40:6789/0 conn(0x7f750c0b1870 legacy=0x7f750c145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.714+0000 7f750bfff700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.714+0000 7f750bfff700 1 monclient: found mon.noname-b 2023-09-30T00:47:57.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.714+0000 7f750bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.714+0000 7f750bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.714+0000 7f750bfff700 1 -- 172.21.15.40:0/698955877 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750c0af8f0 con 0x7f750c0b14a0 2023-09-30T00:47:57.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.714+0000 7f7d85ffb700 1 -- 172.21.15.40:0/4218724934 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895299435 0 0) 0x7f7d880a5b20 con 0x7f7d880a5fb0 2023-09-30T00:47:57.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.714+0000 7f7d85ffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.714+0000 7f7d85ffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.714+0000 7f7d85ffb700 10 monclient(hunting): my global_id is 4311 2023-09-30T00:47:57.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.714+0000 7f7d85ffb700 1 -- 172.21.15.40:0/4218724934 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d70003570 con 0x7f7d880a5fb0 2023-09-30T00:47:57.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.721+0000 7f8c977fe700 1 --2- 172.21.15.40:0/1038578848 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ca00b1830 0x7f8ca00a5fb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009f20 tx=0x7f8c90009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b4e6bf685ebbd02 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.725+0000 7fc662d9d700 1 --2- 172.21.15.40:0/317490228 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc64c004f50 0x7fc64c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.725+0000 7fc662d9d700 10 monclient(hunting): get_auth_request con 0x7fc64c004f50 auth_method 0 2023-09-30T00:47:57.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.725+0000 7fc662d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.725+0000 7fc662d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.725+0000 7fc662d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:47:57.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.726+0000 7fc662d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.726+0000 7fc662d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.726+0000 7fc662d9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:47:57.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.729+0000 7fc662d9d700 10 monclient(hunting): handle_auth_done global_id 4308 payload 293 2023-09-30T00:47:57.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.729+0000 7fc662d9d700 1 -- 172.21.15.40:0/317490228 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640af460 msgr2=0x7fc64c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.729+0000 7fc662d9d700 1 --2- 172.21.15.40:0/317490228 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6640af460 0x7fc64c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.729+0000 7fc662d9d700 1 -- 172.21.15.40:0/317490228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64c003b10 msgr2=0x7fc64c003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.729+0000 7fc662d9d700 1 --2- 172.21.15.40:0/317490228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64c003b10 0x7fc64c003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.729+0000 7fc662d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.729+0000 7fc662d9d700 1 monclient: found mon.b 2023-09-30T00:47:57.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.729+0000 7fc662d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.729+0000 7fc662d9d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.729+0000 7fc662d9d700 1 -- 172.21.15.40:0/317490228 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c003570 con 0x7fc64c004f50 2023-09-30T00:47:57.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.729+0000 7fc662d9d700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.729+0000 7fc662d9d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.731+0000 7fc662d9d700 1 --2- 172.21.15.40:0/317490228 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc64c004f50 0x7fc64c00ba60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00a880 tx=0x7fc66413fca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4504ec9ff88393a5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc66159a700 1 -- 172.21.15.40:0/317490228 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c0070f0 con 0x7fc64c004f50 2023-09-30T00:47:57.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc66159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc66159a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:57.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc66159a700 10 monclient: dump: 2023-09-30T00:47:57.736 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.736 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.736 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.736 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.736 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.736 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.736 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.736 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.736 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.736 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc66159a700 1 -- 172.21.15.40:0/317490228 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65c007250 con 0x7fc64c004f50 2023-09-30T00:47:57.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc668f81700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:57.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc66159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc668f81700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:57.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc66159a700 1 -- 172.21.15.40:0/317490228 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65c007610 con 0x7fc64c004f50 2023-09-30T00:47:57.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc668f81700 4 set_mon_vals no callback set 2023-09-30T00:47:57.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc66159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc66159a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:57.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc66159a700 10 monclient: dump: 2023-09-30T00:47:57.737 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.737 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.737 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.737 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.738 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.738 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.738 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.738 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.738 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.738 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc668f81700 10 monclient: shutdown 2023-09-30T00:47:57.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc668f81700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:57.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc668f81700 1 -- 172.21.15.40:0/317490228 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc64c004f50 msgr2=0x7fc64c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc668f81700 1 --2- 172.21.15.40:0/317490228 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc64c004f50 0x7fc64c00ba60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00a880 tx=0x7fc66413fca0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc662d9d700 1 -- 172.21.15.40:0/317490228 reap_dead start 2023-09-30T00:47:57.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc668f81700 1 -- 172.21.15.40:0/317490228 shutdown_connections 2023-09-30T00:47:57.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.735+0000 7fc668f81700 1 -- 172.21.15.40:0/317490228 >> 172.21.15.40:0/317490228 conn(0x7fc66406af40 msgr2=0x7fc6640631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:57.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.736+0000 7fc668f81700 1 -- 172.21.15.40:0/317490228 shutdown_connections 2023-09-30T00:47:57.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.736+0000 7fc668f81700 1 -- 172.21.15.40:0/317490228 wait complete. 2023-09-30T00:47:57.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.737+0000 7fc668f81700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.737+0000 7fc668f81700 10 monclient: monmap: 2023-09-30T00:47:57.740 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.740 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:57.741 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.737+0000 7fc668f81700 1 librados: starting msgr at 2023-09-30T00:47:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.737+0000 7fc668f81700 1 librados: starting objecter 2023-09-30T00:47:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 1 Processor -- start 2023-09-30T00:47:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 1 -- start start 2023-09-30T00:47:57.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 1 librados: setting wanted keys 2023-09-30T00:47:57.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 1 librados: calling monclient init 2023-09-30T00:47:57.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 10 monclient: init 2023-09-30T00:47:57.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.737+0000 7f96877fe700 1 -- 172.21.15.40:0/4290983236 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9694051d70 con 0x7f96a006beb0 2023-09-30T00:47:57.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.737+0000 7f96877fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.737+0000 7f96877fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:57.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:47:57.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64c004f50 0x7fc66413b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 10 monclient(hunting): picked mon.noname-a con 0x7fc64c004f50 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:57.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66413a630 0x7fc664152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 10 monclient(hunting): picked mon.noname-b con 0x7fc66413a630 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:57.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc664152640 0x7fc664138a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 10 monclient(hunting): picked mon.noname-c con 0x7fc664152640 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66413c8b0 con 0x7fc64c004f50 2023-09-30T00:47:57.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc664068730 con 0x7fc66413a630 2023-09-30T00:47:57.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc664152a70 con 0x7fc664152640 2023-09-30T00:47:57.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.738+0000 7fc668f81700 10 monclient(hunting): authenticate will time out at 5339.039871s 2023-09-30T00:47:57.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.739+0000 7fc661d9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66413a630 0x7fc664152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.739+0000 7fc661d9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66413a630 0x7fc664152100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43312/0 (socket says 172.21.15.40:43312) 2023-09-30T00:47:57.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.739+0000 7fc661d9b700 1 -- 172.21.15.40:0/2670799539 learned_addr learned my addr 172.21.15.40:0/2670799539 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.739+0000 7fc661d9b700 10 monclient(hunting): get_auth_request con 0x7fc66413a630 auth_method 0 2023-09-30T00:47:57.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.739+0000 7fc661d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.739+0000 7fc661d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.739+0000 7fc661d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.740+0000 7fc661d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.740+0000 7fc661d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.740+0000 7fc661d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.744+0000 7fc661d9b700 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2023-09-30T00:47:57.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.744+0000 7fc661d9b700 1 -- 172.21.15.40:0/2670799539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc664152640 msgr2=0x7fc664138a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.744+0000 7fc661d9b700 1 --2- 172.21.15.40:0/2670799539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc664152640 0x7fc664138a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.744+0000 7fc661d9b700 1 -- 172.21.15.40:0/2670799539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64c004f50 msgr2=0x7fc66413b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.744+0000 7fc661d9b700 1 --2- 172.21.15.40:0/2670799539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64c004f50 0x7fc66413b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.744+0000 7fc661d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.744+0000 7fc661d9b700 1 monclient: found mon.noname-b 2023-09-30T00:47:57.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.744+0000 7fc661d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.744+0000 7fc661d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.744+0000 7fc661d9b700 1 -- 172.21.15.40:0/2670799539 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66413aaa0 con 0x7fc66413a630 2023-09-30T00:47:57.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 -- 172.21.15.40:0/4218724934 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2386128388 0 0) 0x7f7d70003570 con 0x7f7d880a5fb0 2023-09-30T00:47:57.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 -- 172.21.15.40:0/4218724934 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d880b1460 msgr2=0x7f7d880b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 --2- 172.21.15.40:0/4218724934 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d880b1460 0x7f7d880b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 -- 172.21.15.40:0/4218724934 >> v1:172.21.15.135:6789/0 conn(0x7f7d8813ac20 legacy=0x7f7d8813b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 monclient: found mon.noname-a-legacy 2023-09-30T00:47:57.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2023-09-30T00:47:57.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 -- 172.21.15.40:0/4218724934 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d880a63c0 con 0x7f7d880a5fb0 2023-09-30T00:47:57.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 -- 172.21.15.40:0/4218724934 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3033000540 0 0) 0x7f7d7c001470 con 0x7f7d880a5fb0 2023-09-30T00:47:57.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.747+0000 7f1b35ffb700 1 -- 172.21.15.40:0/846249853 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1b30053b40 con 0x7f1b400a7610 2023-09-30T00:47:57.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.747+0000 7f1b35ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.747+0000 7f1b35ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:57.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.747+0000 7fc661d9b700 1 --2- 172.21.15.40:0/2670799539 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66413a630 0x7fc664152100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc65800b600 tx=0x7fc6580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.747+0000 7fc6527fc700 10 client.?.objecter ms_handle_connect 0x7fc66413a630 2023-09-30T00:47:57.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.747+0000 7fc6527fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:57.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T00:47:57.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient: dump: 2023-09-30T00:47:57.750 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.750 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.750 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.750 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.750 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.750 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.750 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.751 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2023-09-30T00:47:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 -- 172.21.15.40:0/4218724934 >> v1:172.21.15.40:6789/0 conn(0x7f7d880a5fb0 legacy=0x7f7d88141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:47:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 --2- 172.21.15.40:0/4218724934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d70003a90 0x7f7d70003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient(hunting): picked mon.a con 0x7f7d70003a90 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:57.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 --2- 172.21.15.40:0/4218724934 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d70004810 0x7f7d70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient(hunting): picked mon.c con 0x7f7d70004810 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 --2- 172.21.15.40:0/4218724934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d700052c0 0x7f7d7000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient(hunting): picked mon.b con 0x7f7d700052c0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 -- 172.21.15.40:0/4218724934 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880a74d0 con 0x7f7d70003a90 2023-09-30T00:47:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 -- 172.21.15.40:0/4218724934 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7000c250 con 0x7f7d700052c0 2023-09-30T00:47:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 1 -- 172.21.15.40:0/4218724934 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d70014c20 con 0x7f7d70004810 2023-09-30T00:47:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.746+0000 7f7d85ffb700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.747+0000 7f7d85ffb700 10 monclient(hunting): _finish_auth 0 2023-09-30T00:47:57.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.749+0000 7f7d877fe700 1 --2- 172.21.15.40:0/4218724934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d700052c0 0x7f7d7000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.751+0000 7f7d877fe700 10 monclient(hunting): get_auth_request con 0x7f7d700052c0 auth_method 0 2023-09-30T00:47:57.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.751+0000 7f7d877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.751+0000 7f7d877fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.751+0000 7f7d877fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:47:57.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.754+0000 7f7d877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.754+0000 7f7d877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.754+0000 7f7d877fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:47:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7f7d877fe700 10 monclient(hunting): handle_auth_done global_id 4311 payload 293 2023-09-30T00:47:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7f7d877fe700 1 -- 172.21.15.40:0/4218724934 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d70004810 msgr2=0x7f7d70004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7f7d877fe700 1 --2- 172.21.15.40:0/4218724934 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d70004810 0x7f7d70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7f7d877fe700 1 -- 172.21.15.40:0/4218724934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d70003a90 msgr2=0x7f7d70003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7f7d877fe700 1 --2- 172.21.15.40:0/4218724934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d70003a90 0x7f7d70003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7f7d877fe700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7f7d877fe700 1 monclient: found mon.b 2023-09-30T00:47:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7f7d877fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7f7d877fe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7f7d877fe700 1 -- 172.21.15.40:0/4218724934 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d70003570 con 0x7f7d700052c0 2023-09-30T00:47:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7f7d877fe700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7f7d877fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7f7d8d530700 5 monclient: authenticate success, global_id 4311 2023-09-30T00:47:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7f7d8d530700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:57.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.757+0000 7f7d877fe700 1 --2- 172.21.15.40:0/4218724934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d700052c0 0x7f7d7000ba60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7d80001fc0 tx=0x7f7d80001640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=913a8df3f193450b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7fc6527fc700 1 -- 172.21.15.40:0/2670799539 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65800f510 con 0x7fc66413a630 2023-09-30T00:47:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7fc6527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7fc6527fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7fc6527fc700 10 monclient: dump: 2023-09-30T00:47:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.760 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.760 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7fc6527fc700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7fc6527fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7fc6527fc700 1 -- 172.21.15.40:0/2670799539 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65800f670 con 0x7fc66413a630 2023-09-30T00:47:57.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7fc6527fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7fc6527fc700 1 -- 172.21.15.40:0/2670799539 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65800fa30 con 0x7fc66413a630 2023-09-30T00:47:57.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7fc6527fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7fc6527fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:57.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7fc6527fc700 10 monclient: dump: 2023-09-30T00:47:57.760 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.760 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.761 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.761 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.761 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.761 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.761 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.761 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.761 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.761 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.756+0000 7fc660d99700 4 set_mon_vals no callback set 2023-09-30T00:47:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7fc668f81700 5 monclient: authenticate success, global_id 4348 2023-09-30T00:47:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7fc668f81700 10 monclient: _renew_subs 2023-09-30T00:47:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7fc668f81700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7fc668f81700 1 -- 172.21.15.40:0/2670799539 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc66413ced0 con 0x7fc66413a630 2023-09-30T00:47:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7fc668f81700 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7fc668f81700 10 monclient: _renew_subs 2023-09-30T00:47:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7fc668f81700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7fc668f81700 1 -- 172.21.15.40:0/2670799539 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6641391d0 con 0x7fc66413a630 2023-09-30T00:47:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7fc668f81700 1 librados: init done 2023-09-30T00:47:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.759+0000 7fc668f81700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.759+0000 7fc668f81700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.759+0000 7fc668f81700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.759+0000 7fc668f81700 1 -- 172.21.15.40:0/2670799539 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66413d0f0 con 0x7fc66413a630 2023-09-30T00:47:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7f8c9f7fe700 1 -- 172.21.15.40:0/1038578848 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9000e070 con 0x7f8ca00b1830 2023-09-30T00:47:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7f8c9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7f8c9f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7f8c9f7fe700 10 monclient: dump: 2023-09-30T00:47:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.765 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7f8c9f7fe700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7f8c9f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7f8c9f7fe700 1 -- 172.21.15.40:0/1038578848 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c9000ba40 con 0x7f8ca00b1830 2023-09-30T00:47:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7f8c9f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7f8c9f7fe700 1 -- 172.21.15.40:0/1038578848 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c900092e0 con 0x7f8ca00b1830 2023-09-30T00:47:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7f8c9f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7f8c9f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.758+0000 7f8c9f7fe700 10 monclient: dump: 2023-09-30T00:47:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.766 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.759+0000 7f8ca65cb700 5 monclient: authenticate success, global_id 4277 2023-09-30T00:47:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.759+0000 7f8ca65cb700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.759+0000 7f8ca65cb700 4 set_mon_vals no callback set 2023-09-30T00:47:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.760+0000 7f8ca65cb700 10 monclient: shutdown 2023-09-30T00:47:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.760+0000 7f8ca65cb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.760+0000 7f8ca65cb700 1 -- 172.21.15.40:0/1038578848 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ca00b1830 msgr2=0x7f8ca00a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.760+0000 7f8ca65cb700 1 --2- 172.21.15.40:0/1038578848 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ca00b1830 0x7f8ca00a5fb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009f20 tx=0x7f8c90009840 comp rx=0 tx=0).stop 2023-09-30T00:47:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.763+0000 7f8ca65cb700 1 -- 172.21.15.40:0/1038578848 shutdown_connections 2023-09-30T00:47:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.763+0000 7f8ca65cb700 1 --2- 172.21.15.40:0/1038578848 >> v2:172.21.15.40:3300/0 conn(0x7f8ca013ac20 0x7f8ca013b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.763+0000 7f8ca65cb700 1 --2- 172.21.15.40:0/1038578848 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ca00b1830 0x7f8ca00a5fb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.763+0000 7f8ca65cb700 1 -- 172.21.15.40:0/1038578848 >> 172.21.15.40:0/1038578848 conn(0x7f8ca00631f0 msgr2=0x7f8ca00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.766+0000 7f8ca65cb700 1 -- 172.21.15.40:0/1038578848 shutdown_connections 2023-09-30T00:47:57.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.767+0000 7f8ca65cb700 1 -- 172.21.15.40:0/1038578848 wait complete. 2023-09-30T00:47:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.767+0000 7f8ca65cb700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.767+0000 7f8ca65cb700 10 monclient: monmap: 2023-09-30T00:47:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:57.770 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.768+0000 7f8ca65cb700 1 librados: starting msgr at 2023-09-30T00:47:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.768+0000 7f8ca65cb700 1 librados: starting objecter 2023-09-30T00:47:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.768+0000 7f8ca65cb700 1 Processor -- start 2023-09-30T00:47:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.768+0000 7f8ca65cb700 1 -- start start 2023-09-30T00:47:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.768+0000 7f8ca65cb700 1 librados: setting wanted keys 2023-09-30T00:47:57.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.768+0000 7f8ca65cb700 1 librados: calling monclient init 2023-09-30T00:47:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.768+0000 7f8ca65cb700 10 monclient: init 2023-09-30T00:47:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f7d85ffb700 1 -- 172.21.15.40:0/4218724934 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80001950 con 0x7f7d700052c0 2023-09-30T00:47:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f7d85ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f7d85ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f7d85ffb700 10 monclient: dump: 2023-09-30T00:47:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.771 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.772 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f7d85ffb700 1 -- 172.21.15.40:0/4218724934 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8000c8b0 con 0x7f7d700052c0 2023-09-30T00:47:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f7d85ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f7d85ffb700 1 -- 172.21.15.40:0/4218724934 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8000cc30 con 0x7f7d700052c0 2023-09-30T00:47:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f7d85ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f7d85ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f7d85ffb700 10 monclient: dump: 2023-09-30T00:47:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.773 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.773 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f8ca65cb700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f8ca65cb700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:47:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f8ca65cb700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ca00b1460 0x7f8ca00a93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f8ca65cb700 10 monclient(hunting): picked mon.noname-b con 0x7f8ca00b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f8ca65cb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ca00b1830 0x7f8ca00a8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f8ca65cb700 10 monclient(hunting): picked mon.noname-c con 0x7f8ca00b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.769+0000 7f8ca65cb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ca013ac20 0x7f8ca013d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.770+0000 7f8ca65cb700 10 monclient(hunting): picked mon.noname-a con 0x7f8ca013ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.770+0000 7f8ca65cb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.770+0000 7f8ca65cb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca00aa770 con 0x7f8ca013ac20 2023-09-30T00:47:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.770+0000 7f8ca65cb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.770+0000 7f8ca65cb700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca00aaa50 con 0x7f8ca00b1460 2023-09-30T00:47:57.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.770+0000 7f8ca65cb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.770+0000 7f8ca65cb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca013f3f0 con 0x7f8ca00b1830 2023-09-30T00:47:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.770+0000 7f8ca65cb700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.770+0000 7f8ca65cb700 10 monclient(hunting): authenticate will time out at 5339.071056s 2023-09-30T00:47:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.772+0000 7f8c977fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ca00b1830 0x7f8ca00a8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.773+0000 7f7d8d530700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:57.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.773+0000 7f7d8d530700 4 set_mon_vals no callback set 2023-09-30T00:47:57.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.773+0000 7f8c977fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ca00b1830 0x7f8ca00a8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36664/0 (socket says 172.21.15.40:36664) 2023-09-30T00:47:57.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.773+0000 7f8c977fe700 1 -- 172.21.15.40:0/670833357 learned_addr learned my addr 172.21.15.40:0/670833357 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.773+0000 7f8c977fe700 10 monclient(hunting): get_auth_request con 0x7f8ca00b1830 auth_method 0 2023-09-30T00:47:57.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.773+0000 7f8c977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.773+0000 7f8c977fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.773+0000 7f8c977fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.774+0000 7f7d8d530700 10 monclient: shutdown 2023-09-30T00:47:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.774+0000 7f7d8d530700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.774+0000 7f7d8d530700 1 -- 172.21.15.40:0/4218724934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d700052c0 msgr2=0x7f7d7000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.774+0000 7f7d8d530700 1 --2- 172.21.15.40:0/4218724934 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d700052c0 0x7f7d7000ba60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7d80001fc0 tx=0x7f7d80001640 comp rx=0 tx=0).stop 2023-09-30T00:47:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.774+0000 7f7d877fe700 1 -- 172.21.15.40:0/4218724934 reap_dead start 2023-09-30T00:47:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.774+0000 7f7d8d530700 1 -- 172.21.15.40:0/4218724934 shutdown_connections 2023-09-30T00:47:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.774+0000 7f7d8d530700 1 -- 172.21.15.40:0/4218724934 >> 172.21.15.40:0/4218724934 conn(0x7f7d880631f0 msgr2=0x7f7d880635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:57.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.773+0000 7f750bfff700 1 --2- 172.21.15.40:0/698955877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f750c0b14a0 0x7f750c0afe10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f75000099e0 tx=0x7f7500009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d69d62fc1328fd79 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.774+0000 7f7d8d530700 1 -- 172.21.15.40:0/4218724934 shutdown_connections 2023-09-30T00:47:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.774+0000 7f7d8d530700 1 -- 172.21.15.40:0/4218724934 wait complete. 2023-09-30T00:47:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.775+0000 7f8c977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.775+0000 7f8c977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.775+0000 7f8c977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.775+0000 7f7d8d530700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.775+0000 7f7d8d530700 10 monclient: monmap: 2023-09-30T00:47:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.780 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:57.781 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.776+0000 7f7d8d530700 1 librados: starting msgr at 2023-09-30T00:47:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.776+0000 7f7d8d530700 1 librados: starting objecter 2023-09-30T00:47:57.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.776+0000 7f7d8d530700 1 Processor -- start 2023-09-30T00:47:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.776+0000 7f7d8d530700 1 -- start start 2023-09-30T00:47:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.776+0000 7f7d8d530700 1 librados: setting wanted keys 2023-09-30T00:47:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.776+0000 7f7d8d530700 1 librados: calling monclient init 2023-09-30T00:47:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.776+0000 7f7d8d530700 10 monclient: init 2023-09-30T00:47:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:47:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d700052c0 0x7f7d8806d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 10 monclient(hunting): picked mon.noname-a con 0x7f7d700052c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d8806dc10 0x7f7d8806a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 10 monclient(hunting): picked mon.noname-b con 0x7f7d8806dc10 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:57.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d8806ae80 0x7f7d8814cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 10 monclient(hunting): picked mon.noname-c con 0x7f7d8806ae80 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8806b2f0 con 0x7f7d700052c0 2023-09-30T00:47:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8806e1c0 con 0x7f7d8806dc10 2023-09-30T00:47:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880ae510 con 0x7f7d8806ae80 2023-09-30T00:47:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.777+0000 7f7d8d530700 10 monclient(hunting): authenticate will time out at 5339.078427s 2023-09-30T00:47:57.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.780+0000 7f8c977fe700 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2023-09-30T00:47:57.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.780+0000 7f8c977fe700 1 -- 172.21.15.40:0/670833357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ca00b1460 msgr2=0x7f8ca00a93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.780+0000 7f8c977fe700 1 --2- 172.21.15.40:0/670833357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ca00b1460 0x7f8ca00a93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.780+0000 7f8c977fe700 1 -- 172.21.15.40:0/670833357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ca013ac20 msgr2=0x7f8ca013d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.780+0000 7f8c977fe700 1 --2- 172.21.15.40:0/670833357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ca013ac20 0x7f8ca013d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.780+0000 7f8c977fe700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.780+0000 7f8c977fe700 1 monclient: found mon.noname-c 2023-09-30T00:47:57.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.780+0000 7f8c977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.780+0000 7f8c977fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.780+0000 7f8c977fe700 1 -- 172.21.15.40:0/670833357 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c90009ae0 con 0x7f8ca00b1830 2023-09-30T00:47:57.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8330871700 1 -- 172.21.15.40:0/2639070419 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832800b7e0 con 0x7f832c145d30 2023-09-30T00:47:57.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8330871700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8330871700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:57.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8330871700 10 monclient: dump: 2023-09-30T00:47:57.786 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.786 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.786 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.786 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.786 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.786 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.786 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.786 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.786 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.787 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8330871700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8330871700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8330871700 1 -- 172.21.15.40:0/2639070419 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832800b940 con 0x7f832c145d30 2023-09-30T00:47:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8330871700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8330871700 1 -- 172.21.15.40:0/2639070419 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832800bd00 con 0x7f832c145d30 2023-09-30T00:47:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8330871700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8330871700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8330871700 10 monclient: dump: 2023-09-30T00:47:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.787 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.788 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.788 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.788 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.788 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.788 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.788 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.788 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8333ad7700 5 monclient: authenticate success, global_id 4305 2023-09-30T00:47:57.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8333ad7700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:57.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.783+0000 7f8333ad7700 4 set_mon_vals no callback set 2023-09-30T00:47:57.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.784+0000 7f8333ad7700 10 monclient: shutdown 2023-09-30T00:47:57.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.784+0000 7f8333ad7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:57.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.784+0000 7f8333ad7700 1 -- 172.21.15.40:0/2639070419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f832c145d30 msgr2=0x7f832c1461b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.784+0000 7f8333ad7700 1 --2- 172.21.15.40:0/2639070419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f832c145d30 0x7f832c1461b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f83280099b0 tx=0x7f8328009250 comp rx=0 tx=0).stop 2023-09-30T00:47:57.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.784+0000 7f8333ad7700 1 -- 172.21.15.40:0/2639070419 shutdown_connections 2023-09-30T00:47:57.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.784+0000 7f8333ad7700 1 --2- 172.21.15.40:0/2639070419 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f832c145d30 0x7f832c1461b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.784+0000 7f8333ad7700 1 --2- 172.21.15.40:0/2639070419 >> v2:172.21.15.135:3300/0 conn(0x7f832c06f0f0 0x7f832c06e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.784+0000 7f8333ad7700 1 -- 172.21.15.40:0/2639070419 >> 172.21.15.40:0/2639070419 conn(0x7f832c067bf0 msgr2=0x7f832c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:57.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.784+0000 7f8333ad7700 1 -- 172.21.15.40:0/2639070419 shutdown_connections 2023-09-30T00:47:57.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.784+0000 7f8333ad7700 1 -- 172.21.15.40:0/2639070419 wait complete. 2023-09-30T00:47:57.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.785+0000 7f8333ad7700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.785+0000 7f8333ad7700 10 monclient: monmap: 2023-09-30T00:47:57.790 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.790 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:57.790 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:57.791 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:57.791 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.791 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:57.791 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:57.791 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:57.791 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:57.791 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.786+0000 7f8333ad7700 1 librados: starting msgr at 2023-09-30T00:47:57.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.786+0000 7f8333ad7700 1 librados: starting objecter 2023-09-30T00:47:57.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.786+0000 7f8333ad7700 1 Processor -- start 2023-09-30T00:47:57.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.786+0000 7f8333ad7700 1 -- start start 2023-09-30T00:47:57.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.786+0000 7f8333ad7700 1 librados: setting wanted keys 2023-09-30T00:47:57.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.786+0000 7f8333ad7700 1 librados: calling monclient init 2023-09-30T00:47:57.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.786+0000 7f8333ad7700 10 monclient: init 2023-09-30T00:47:57.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:47:57.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f832c06f0f0 0x7f832c147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 10 monclient(hunting): picked mon.noname-c con 0x7f832c06f0f0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f832c0b0f50 0x7f832c0a9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 10 monclient(hunting): picked mon.noname-a con 0x7f832c0b0f50 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:57.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f832c145d30 0x7f832c12f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 10 monclient(hunting): picked mon.noname-b con 0x7f832c145d30 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:57.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c12ecc0 con 0x7f832c0b0f50 2023-09-30T00:47:57.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c1475c0 con 0x7f832c145d30 2023-09-30T00:47:57.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832c0aa1a0 con 0x7f832c06f0f0 2023-09-30T00:47:57.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.787+0000 7f8333ad7700 10 monclient(hunting): authenticate will time out at 5339.088707s 2023-09-30T00:47:57.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f750affd700 1 -- 172.21.15.40:0/698955877 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750000b7e0 con 0x7f750c0b14a0 2023-09-30T00:47:57.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f750affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f750affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:57.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f750affd700 10 monclient: dump: 2023-09-30T00:47:57.794 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.794 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.794 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.794 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.794 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.794 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.794 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.794 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.795 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.795 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f750affd700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f750affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f750affd700 1 -- 172.21.15.40:0/698955877 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750000b940 con 0x7f750c0b14a0 2023-09-30T00:47:57.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f7512867700 5 monclient: authenticate success, global_id 4280 2023-09-30T00:47:57.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f7512867700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:57.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f750affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f750affd700 1 -- 172.21.15.40:0/698955877 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750000bce0 con 0x7f750c0b14a0 2023-09-30T00:47:57.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f750affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f750affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:57.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f750affd700 10 monclient: dump: 2023-09-30T00:47:57.796 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.796 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.796 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.796 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.796 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.796 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.796 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.796 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.796 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.796 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f7512867700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.791+0000 7f7512867700 4 set_mon_vals no callback set 2023-09-30T00:47:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.792+0000 7f7512867700 10 monclient: shutdown 2023-09-30T00:47:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.792+0000 7f7512867700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.792+0000 7f7512867700 1 -- 172.21.15.40:0/698955877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f750c0b14a0 msgr2=0x7f750c0afe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.792+0000 7f7512867700 1 --2- 172.21.15.40:0/698955877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f750c0b14a0 0x7f750c0afe10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f75000099e0 tx=0x7f7500009250 comp rx=0 tx=0).stop 2023-09-30T00:47:57.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.794+0000 7f8331072700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f832c0b0f50 0x7f832c0a9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.794+0000 7f8c977fe700 1 --2- 172.21.15.40:0/670833357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ca00b1830 0x7f8ca00a8640 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000b6c0 tx=0x7f8c90017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.794+0000 7f8c9cff9700 10 client.?.objecter ms_handle_connect 0x7f8ca00b1830 2023-09-30T00:47:57.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.794+0000 7f8c9cff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:57.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.795+0000 7f7d877fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d8806ae80 0x7f7d8814cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.795+0000 7f7512867700 1 -- 172.21.15.40:0/698955877 shutdown_connections 2023-09-30T00:47:57.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.795+0000 7f7512867700 1 --2- 172.21.15.40:0/698955877 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f750c0b14a0 0x7f750c0afe10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.795+0000 7f7512867700 1 -- 172.21.15.40:0/698955877 >> 172.21.15.40:0/698955877 conn(0x7f750c0631f0 msgr2=0x7f750c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:57.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.795+0000 7f7d86ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d700052c0 0x7f7d8806d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.796+0000 7f7512867700 1 -- 172.21.15.40:0/698955877 shutdown_connections 2023-09-30T00:47:57.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.796+0000 7f7512867700 1 -- 172.21.15.40:0/698955877 wait complete. 2023-09-30T00:47:57.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.796+0000 7f7d86ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d700052c0 0x7f7d8806d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56670/0 (socket says 172.21.15.40:56670) 2023-09-30T00:47:57.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.796+0000 7f7d86ffd700 1 -- 172.21.15.40:0/4262473729 learned_addr learned my addr 172.21.15.40:0/4262473729 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.796+0000 7f7d86ffd700 10 monclient(hunting): get_auth_request con 0x7f7d700052c0 auth_method 0 2023-09-30T00:47:57.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.796+0000 7f7d86ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.796+0000 7f7d86ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.796+0000 7f7d86ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f7d877fe700 10 monclient(hunting): get_auth_request con 0x7f7d8806ae80 auth_method 0 2023-09-30T00:47:57.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f7d877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f7d877fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f7d877fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f7512867700 10 monclient: build_initial_monmap 2023-09-30T00:47:57.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f7512867700 10 monclient: monmap: 2023-09-30T00:47:57.801 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:57.801 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:57.801 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:57.801 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:57.801 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:57.801 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:57.802 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:57.802 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:57.802 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:57.802 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f7512867700 1 librados: starting msgr at 2023-09-30T00:47:57.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f7512867700 1 librados: starting objecter 2023-09-30T00:47:57.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f7512867700 1 Processor -- start 2023-09-30T00:47:57.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f8331072700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f832c0b0f50 0x7f832c0a9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56676/0 (socket says 172.21.15.40:56676) 2023-09-30T00:47:57.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f8331072700 1 -- 172.21.15.40:0/2098221181 learned_addr learned my addr 172.21.15.40:0/2098221181 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f8331072700 10 monclient(hunting): get_auth_request con 0x7f832c0b0f50 auth_method 0 2023-09-30T00:47:57.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f8331072700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f8331072700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f8331072700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.798+0000 7f7512867700 1 -- start start 2023-09-30T00:47:57.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.798+0000 7f7512867700 1 librados: setting wanted keys 2023-09-30T00:47:57.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.798+0000 7f7512867700 1 librados: calling monclient init 2023-09-30T00:47:57.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.798+0000 7f7512867700 10 monclient: init 2023-09-30T00:47:57.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.798+0000 7f7512867700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:57.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.798+0000 7f7512867700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:47:57.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f7512867700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f750c06dd20 0x7f750c06c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f7512867700 10 monclient(hunting): picked mon.noname-c con 0x7f750c06dd20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:57.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f7512867700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f750c0b14a0 0x7f750c0b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f7512867700 10 monclient(hunting): picked mon.noname-a con 0x7f750c0b14a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:57.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f7512867700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f750c0b1870 0x7f750c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f7512867700 10 monclient(hunting): picked mon.noname-b con 0x7f750c0b1870 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:57.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f7512867700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f7512867700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c0b4c00 con 0x7f750c0b14a0 2023-09-30T00:47:57.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f7512867700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f7512867700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c11d380 con 0x7f750c0b1870 2023-09-30T00:47:57.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f7512867700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:57.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f7512867700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750c11d5f0 con 0x7f750c06dd20 2023-09-30T00:47:57.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f7512867700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:57.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f7512867700 10 monclient(hunting): authenticate will time out at 5339.100098s 2023-09-30T00:47:57.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.798+0000 7f8331873700 1 --2- 172.21.15.40:0/2098221181 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f832c06f0f0 0x7f832c147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f8331072700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f8331072700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.799+0000 7f8331072700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.800+0000 7f8331873700 10 monclient(hunting): get_auth_request con 0x7f832c06f0f0 auth_method 0 2023-09-30T00:47:57.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.800+0000 7f8331873700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.800+0000 7f8331873700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.800+0000 7f8331873700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f7d86ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f7d86ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.797+0000 7f7d86ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.805+0000 7f8331072700 10 monclient(hunting): handle_auth_done global_id 4317 payload 824 2023-09-30T00:47:57.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.805+0000 7f8331072700 1 -- 172.21.15.40:0/2098221181 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f832c06f0f0 msgr2=0x7f832c147080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.805+0000 7f8331072700 1 --2- 172.21.15.40:0/2098221181 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f832c06f0f0 0x7f832c147080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.805+0000 7f8331072700 1 -- 172.21.15.40:0/2098221181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f832c145d30 msgr2=0x7f832c12f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.805+0000 7f8331072700 1 --2- 172.21.15.40:0/2098221181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f832c145d30 0x7f832c12f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.805+0000 7f8331072700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.805+0000 7f8331072700 1 monclient: found mon.noname-a 2023-09-30T00:47:57.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.805+0000 7f8331072700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.805+0000 7f8331072700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:57.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.805+0000 7f8331072700 1 -- 172.21.15.40:0/2098221181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83280094f0 con 0x7f832c0b0f50 2023-09-30T00:47:57.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.806+0000 7f7d877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.806+0000 7f7d877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.806+0000 7f7d877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.812+0000 7f7d86ffd700 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2023-09-30T00:47:57.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.812+0000 7f7d86ffd700 1 -- 172.21.15.40:0/4262473729 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d8806ae80 msgr2=0x7f7d8814cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.812+0000 7f7d86ffd700 1 --2- 172.21.15.40:0/4262473729 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d8806ae80 0x7f7d8814cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.812+0000 7f7d86ffd700 1 -- 172.21.15.40:0/4262473729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d8806dc10 msgr2=0x7f7d8806a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.812+0000 7f7d86ffd700 1 --2- 172.21.15.40:0/4262473729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d8806dc10 0x7f7d8806a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.812+0000 7f7d86ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.812+0000 7f7d86ffd700 1 monclient: found mon.noname-a 2023-09-30T00:47:57.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.812+0000 7f7d86ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.812+0000 7f7d86ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:57.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.812+0000 7f7d86ffd700 1 -- 172.21.15.40:0/4262473729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8806c700 con 0x7f7d700052c0 2023-09-30T00:47:57.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.822+0000 7f750bfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f750c06dd20 0x7f750c06c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.825+0000 7f750bfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f750c06dd20 0x7f750c06c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36680/0 (socket says 172.21.15.40:36680) 2023-09-30T00:47:57.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.825+0000 7f750bfff700 1 -- 172.21.15.40:0/229920520 learned_addr learned my addr 172.21.15.40:0/229920520 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:57.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.825+0000 7f750bfff700 10 monclient(hunting): get_auth_request con 0x7f750c06dd20 auth_method 0 2023-09-30T00:47:57.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.825+0000 7f750bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:57.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.825+0000 7f750bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:57.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.825+0000 7f750bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:57.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.828+0000 7f750bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:57.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.828+0000 7f750bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:57.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.828+0000 7f750bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.833+0000 7f750bfff700 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2023-09-30T00:47:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.833+0000 7f750bfff700 1 -- 172.21.15.40:0/229920520 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f750c0b1870 msgr2=0x7f750c0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.833+0000 7f750bfff700 1 --2- 172.21.15.40:0/229920520 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f750c0b1870 0x7f750c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.833+0000 7f750bfff700 1 -- 172.21.15.40:0/229920520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f750c0b14a0 msgr2=0x7f750c0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:57.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.833+0000 7f750bfff700 1 --2- 172.21.15.40:0/229920520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f750c0b14a0 0x7f750c0b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.833+0000 7f750bfff700 10 monclient: _finish_hunting 0 2023-09-30T00:47:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.833+0000 7f750bfff700 1 monclient: found mon.noname-c 2023-09-30T00:47:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.833+0000 7f750bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.833+0000 7f750bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.833+0000 7f750bfff700 1 -- 172.21.15.40:0/229920520 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75000094f0 con 0x7f750c06dd20 2023-09-30T00:47:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.839+0000 7f750bfff700 1 --2- 172.21.15.40:0/229920520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f750c06dd20 0x7f750c06c520 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7500009930 tx=0x7f7500016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.839+0000 7f74f3fff700 10 client.?.objecter ms_handle_connect 0x7f750c06dd20 2023-09-30T00:47:57.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.839+0000 7f74f3fff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:57.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.862+0000 7f8331072700 1 --2- 172.21.15.40:0/2098221181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f832c0b0f50 0x7f832c0a9c60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f831c00b810 tx=0x7f831c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.862+0000 7f8321ffb700 10 client.?.objecter ms_handle_connect 0x7f832c0b0f50 2023-09-30T00:47:57.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.862+0000 7f8321ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:57.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.862+0000 7f7d86ffd700 1 --2- 172.21.15.40:0/4262473729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d700052c0 0x7f7d8806d5b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800b580 tx=0x7f7d780091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.862+0000 7f7d6effd700 10 client.?.objecter ms_handle_connect 0x7f7d700052c0 2023-09-30T00:47:57.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.862+0000 7f7d6effd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:57.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.914+0000 7fc6527fc700 1 -- 172.21.15.40:0/2670799539 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fc658024430 con 0x7fc66413a630 2023-09-30T00:47:57.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.914+0000 7fc6527fc700 1 --2- 172.21.15.40:0/2670799539 >> v2:172.21.15.135:6800/116426 conn(0x7fc644068960 0x7fc64406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:57.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.914+0000 7fc6527fc700 1 -- 172.21.15.40:0/2670799539 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fc6580844c0 con 0x7fc66413a630 2023-09-30T00:47:57.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.914+0000 7fc6527fc700 10 client.4348.objecter ms_dispatch 0x7fc664001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:57.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.914+0000 7fc6527fc700 3 client.4348.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:57.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.914+0000 7fc6527fc700 3 client.4348.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:57.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.915+0000 7fc6527fc700 20 client.4348.objecter dump_active .. 0 homeless 2023-09-30T00:47:57.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.915+0000 7fc66259c700 1 --2- 172.21.15.40:0/2670799539 >> v2:172.21.15.135:6800/116426 conn(0x7fc644068960 0x7fc64406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:57.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.915+0000 7fc66259c700 10 monclient: get_auth_request con 0x7fc644068960 auth_method 0 2023-09-30T00:47:57.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.915+0000 7fc6527fc700 10 client.4348.objecter ms_handle_connect 0x7fc644068960 2023-09-30T00:47:57.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.915+0000 7fc66259c700 1 --2- 172.21.15.40:0/2670799539 >> v2:172.21.15.135:6800/116426 conn(0x7fc644068960 0x7fc64406ae10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc654009890 tx=0x7fc654006d40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8c9cff9700 1 -- 172.21.15.40:0/670833357 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c90012a10 con 0x7f8ca00b1830 2023-09-30T00:47:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8c9cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8c9cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8c9cff9700 10 monclient: dump: 2023-09-30T00:47:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.922 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.923 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.923 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.923 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.923 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.923 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.923 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8c9cff9700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8c9cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8c9cff9700 1 -- 172.21.15.40:0/670833357 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c90012b70 con 0x7f8ca00b1830 2023-09-30T00:47:57.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8c9cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8c9cff9700 1 -- 172.21.15.40:0/670833357 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c9001e430 con 0x7f8ca00b1830 2023-09-30T00:47:57.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8c9cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8c9cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:57.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8c9cff9700 10 monclient: dump: 2023-09-30T00:47:57.924 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.924 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.924 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.924 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.925 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8ca65cb700 5 monclient: authenticate success, global_id 4283 2023-09-30T00:47:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8ca65cb700 10 monclient: _renew_subs 2023-09-30T00:47:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8ca65cb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8ca65cb700 1 -- 172.21.15.40:0/670833357 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca013de30 con 0x7f8ca00b1830 2023-09-30T00:47:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8ca65cb700 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8ca65cb700 10 monclient: _renew_subs 2023-09-30T00:47:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8ca65cb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8ca65cb700 1 -- 172.21.15.40:0/670833357 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca013e2d0 con 0x7f8ca00b1830 2023-09-30T00:47:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f8ca65cb700 1 librados: init done 2023-09-30T00:47:57.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.922+0000 7f8c9effd700 4 set_mon_vals no callback set 2023-09-30T00:47:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f74f3fff700 1 -- 172.21.15.40:0/229920520 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7500011d90 con 0x7f750c06dd20 2023-09-30T00:47:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.921+0000 7f74f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.922+0000 7f74f3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.922+0000 7f74f3fff700 10 monclient: dump: 2023-09-30T00:47:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.927 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.928 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.928 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.928 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.928 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.928 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.922+0000 7f74f3fff700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.922+0000 7f74f3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.922+0000 7f7512867700 5 monclient: authenticate success, global_id 4292 2023-09-30T00:47:57.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.922+0000 7f7512867700 10 monclient: _renew_subs 2023-09-30T00:47:57.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.922+0000 7f7512867700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.922+0000 7f7512867700 1 -- 172.21.15.40:0/229920520 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750c0b4810 con 0x7f750c06dd20 2023-09-30T00:47:57.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.922+0000 7f7512867700 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.922+0000 7f7512867700 10 monclient: _renew_subs 2023-09-30T00:47:57.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.922+0000 7f7512867700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.922+0000 7f7512867700 1 -- 172.21.15.40:0/229920520 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750c1554d0 con 0x7f750c06dd20 2023-09-30T00:47:57.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.922+0000 7f7512867700 1 librados: init done 2023-09-30T00:47:57.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.923+0000 7f8ca65cb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.923+0000 7f8ca65cb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.923+0000 7f8ca65cb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.923+0000 7f8ca65cb700 1 -- 172.21.15.40:0/670833357 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c840050b0 con 0x7f8ca00b1830 2023-09-30T00:47:57.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.924+0000 7f7512867700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.924+0000 7f7512867700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.924+0000 7f7512867700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:57.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.924+0000 7f7512867700 1 -- 172.21.15.40:0/229920520 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f750c1554d0 con 0x7f750c06dd20 2023-09-30T00:47:57.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.924+0000 7f74f3fff700 1 -- 172.21.15.40:0/229920520 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7500020070 con 0x7f750c06dd20 2023-09-30T00:47:57.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.924+0000 7f74f3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.928+0000 7f74f3fff700 1 -- 172.21.15.40:0/229920520 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750001b660 con 0x7f750c06dd20 2023-09-30T00:47:57.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.928+0000 7f74f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.928+0000 7f74f3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:57.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.928+0000 7f74f3fff700 10 monclient: dump: 2023-09-30T00:47:57.931 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.931 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.931 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.931 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.932 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.932 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.932 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.932 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.932 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.932 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.932+0000 7f750a7fc700 4 set_mon_vals no callback set 2023-09-30T00:47:57.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.950+0000 7f7d6effd700 1 -- 172.21.15.40:0/4262473729 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7800f510 con 0x7f7d700052c0 2023-09-30T00:47:57.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.950+0000 7f7d6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.950+0000 7f7d6effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:57.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.950+0000 7f7d6effd700 10 monclient: dump: 2023-09-30T00:47:57.952 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.952 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.952 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.952 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.952 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.952 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.952 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.952 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.952 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.952 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.950+0000 7f7d6effd700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.950+0000 7f7d6effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.950+0000 7f7d6effd700 1 -- 172.21.15.40:0/4262473729 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7800f670 con 0x7f7d700052c0 2023-09-30T00:47:57.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.950+0000 7f7d6effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.950+0000 7f7d6effd700 1 -- 172.21.15.40:0/4262473729 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7800fa30 con 0x7f7d700052c0 2023-09-30T00:47:57.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.950+0000 7f7d6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.950+0000 7f7d6effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:57.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.950+0000 7f7d6effd700 10 monclient: dump: 2023-09-30T00:47:57.953 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.953 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.953 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.953 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.954 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.954 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.954 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.954 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.954 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.954 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.951+0000 7f7d8d530700 5 monclient: authenticate success, global_id 4314 2023-09-30T00:47:57.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.951+0000 7f7d8d530700 10 monclient: _renew_subs 2023-09-30T00:47:57.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.951+0000 7f7d8d530700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:57.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.951+0000 7f7d8d530700 1 -- 172.21.15.40:0/4262473729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8806e6e0 con 0x7f7d700052c0 2023-09-30T00:47:57.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.951+0000 7f7d857fa700 4 set_mon_vals no callback set 2023-09-30T00:47:57.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.951+0000 7f7d8d530700 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.951+0000 7f7d8d530700 10 monclient: _renew_subs 2023-09-30T00:47:57.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.951+0000 7f7d8d530700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:57.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.951+0000 7f7d8d530700 1 -- 172.21.15.40:0/4262473729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8814d560 con 0x7f7d700052c0 2023-09-30T00:47:57.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.951+0000 7f7d8d530700 1 librados: init done 2023-09-30T00:47:57.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.953+0000 7f7d8d530700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.953+0000 7f7d8d530700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.953+0000 7f7d8d530700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:57.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.953+0000 7f7d8d530700 1 -- 172.21.15.40:0/4262473729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d880a8ad0 con 0x7f7d700052c0 2023-09-30T00:47:57.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.956+0000 7f8321ffb700 1 -- 172.21.15.40:0/2098221181 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c00f5f0 con 0x7f832c0b0f50 2023-09-30T00:47:57.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.956+0000 7f8321ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.956+0000 7f8321ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:57.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.956+0000 7f8321ffb700 10 monclient: dump: 2023-09-30T00:47:57.957 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.957 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.957 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.958 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.958 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.958 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.958 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.958 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.958 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.958 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.956+0000 7f8321ffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:57.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.956+0000 7f8321ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:57.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.956+0000 7f8321ffb700 1 -- 172.21.15.40:0/2098221181 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831c00f770 con 0x7f832c0b0f50 2023-09-30T00:47:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.956+0000 7f8333ad7700 5 monclient: authenticate success, global_id 4317 2023-09-30T00:47:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.956+0000 7f8321ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.956+0000 7f8321ffb700 1 -- 172.21.15.40:0/2098221181 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c00fb30 con 0x7f832c0b0f50 2023-09-30T00:47:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.956+0000 7f8333ad7700 10 monclient: _renew_subs 2023-09-30T00:47:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.956+0000 7f8333ad7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:57.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.956+0000 7f8333ad7700 1 -- 172.21.15.40:0/2098221181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832c12f720 con 0x7f832c0b0f50 2023-09-30T00:47:57.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.957+0000 7f8333ad7700 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:57.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.957+0000 7f8333ad7700 10 monclient: _renew_subs 2023-09-30T00:47:57.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.957+0000 7f8333ad7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:57.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.957+0000 7f8333ad7700 1 -- 172.21.15.40:0/2098221181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832c0a6f40 con 0x7f832c0b0f50 2023-09-30T00:47:57.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.957+0000 7f8333ad7700 1 librados: init done 2023-09-30T00:47:57.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.957+0000 7f8321ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:57.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.957+0000 7f8321ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:57.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.957+0000 7f8321ffb700 10 monclient: dump: 2023-09-30T00:47:57.960 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:57.960 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:57.961 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.961 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:57.961 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:57.961 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:57.961 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:57.961 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:57.961 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:57.961 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:57.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.958+0000 7f8323fff700 4 set_mon_vals no callback set 2023-09-30T00:47:57.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.960+0000 7f8333ad7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.960+0000 7f8333ad7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:57.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.960+0000 7f8333ad7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:57.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.960+0000 7f8333ad7700 1 -- 172.21.15.40:0/2098221181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830c0050b0 con 0x7f832c0b0f50 2023-09-30T00:47:57.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.973+0000 7fa9a3335700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T00:47:57.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.973+0000 7fa9a3335700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T00:47:57.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.973+0000 7fa9a3335700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:57.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:57.973+0000 7fa9a3335700 1 -- 172.21.15.40:0/891003756 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fa99c15e150 con 0x7fa99c0b0f50 2023-09-30T00:47:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.019+0000 7f1b481c1700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:47:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.019+0000 7f1b481c1700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:47:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.019+0000 7f1b481c1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:58.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.019+0000 7f1b481c1700 1 -- 172.21.15.40:0/846249853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1b40138680 con 0x7f1b400a7610 2023-09-30T00:47:58.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.043+0000 7fc6527fc700 1 -- 172.21.15.40:0/2670799539 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc658051e90 con 0x7fc66413a630 2023-09-30T00:47:58.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.043+0000 7fc6527fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:58.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.043+0000 7fc6527fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:58.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.049+0000 7f96a61d1700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:47:58.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.049+0000 7f96a61d1700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:47:58.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.049+0000 7f96a61d1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:58.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.049+0000 7f96a61d1700 1 -- 172.21.15.40:0/4290983236 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f96a00af410 con 0x7f96a006beb0 2023-09-30T00:47:58.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.072+0000 7f74f3fff700 1 -- 172.21.15.40:0/229920520 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7500022030 con 0x7f750c06dd20 2023-09-30T00:47:58.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.072+0000 7f74f3fff700 1 --2- 172.21.15.40:0/229920520 >> v2:172.21.15.135:6800/116426 conn(0x7f74f8068890 0x7f74f806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:58.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.072+0000 7f74f3fff700 1 -- 172.21.15.40:0/229920520 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7500086120 con 0x7f750c06dd20 2023-09-30T00:47:58.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.072+0000 7f74f3fff700 10 client.4292.objecter ms_dispatch 0x7f750c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:58.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.072+0000 7f74f3fff700 3 client.4292.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.072+0000 7f74f3fff700 3 client.4292.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.073+0000 7f74f3fff700 20 client.4292.objecter dump_active .. 0 homeless 2023-09-30T00:47:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.073+0000 7f750b7fe700 1 --2- 172.21.15.40:0/229920520 >> v2:172.21.15.135:6800/116426 conn(0x7f74f8068890 0x7f74f806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.073+0000 7f750b7fe700 10 monclient: get_auth_request con 0x7f74f8068890 auth_method 0 2023-09-30T00:47:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.073+0000 7f74f3fff700 10 client.4292.objecter ms_handle_connect 0x7f74f8068890 2023-09-30T00:47:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.073+0000 7f750b7fe700 1 --2- 172.21.15.40:0/229920520 >> v2:172.21.15.135:6800/116426 conn(0x7f74f8068890 0x7f74f806ad40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f750c06d5f0 tx=0x7f74fc006c40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:58.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.129+0000 7fa9917fa700 1 -- 172.21.15.40:0/891003756 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa99c15e150 con 0x7fa99c0b0f50 2023-09-30T00:47:58.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.129+0000 7fa9917fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T00:47:58.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.129+0000 7fa9917fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:58.131 INFO:teuthology.orchestra.run.smithi040.stdout:64424509445 2023-09-30T00:47:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 10 librados: watch_flush enter 2023-09-30T00:47:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 10 librados: watch_flush exit 2023-09-30T00:47:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 20 client.4345.objecter shutdown clearing up homeless session... 2023-09-30T00:47:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 10 client.4345.objecter successfully canceled tick 2023-09-30T00:47:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 1 -- 172.21.15.40:0/891003756 >> v2:172.21.15.135:6800/116426 conn(0x7fa980068960 msgr2=0x7fa98006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 1 --2- 172.21.15.40:0/891003756 >> v2:172.21.15.135:6800/116426 conn(0x7fa980068960 0x7fa98006ae10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa994009b90 tx=0x7fa994018000 comp rx=0 tx=0).stop 2023-09-30T00:47:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 10 monclient: shutdown 2023-09-30T00:47:58.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:58.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 1 -- 172.21.15.40:0/891003756 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c0b0f50 msgr2=0x7fa99c15e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 1 --2- 172.21.15.40:0/891003756 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c0b0f50 0x7fa99c15e970 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00b5e0 tx=0x7fa98c0090f0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 1 -- 172.21.15.40:0/891003756 shutdown_connections 2023-09-30T00:47:58.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 1 --2- 172.21.15.40:0/891003756 >> v2:172.21.15.135:6800/116426 conn(0x7fa980068960 0x7fa98006ae10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 1 --2- 172.21.15.40:0/891003756 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa99c0b0f50 0x7fa99c15e970 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 1 --2- 172.21.15.40:0/891003756 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa99c06e980 0x7fa99c077f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 1 --2- 172.21.15.40:0/891003756 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa99c06f840 0x7fa99c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 1 -- 172.21.15.40:0/891003756 >> 172.21.15.40:0/891003756 conn(0x7fa99c067bf0 msgr2=0x7fa99c0afa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:58.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 1 -- 172.21.15.40:0/891003756 shutdown_connections 2023-09-30T00:47:58.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 1 -- 172.21.15.40:0/891003756 wait complete. 2023-09-30T00:47:58.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.132+0000 7fa9a3335700 1 librados: shutdown 2023-09-30T00:47:58.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.148+0000 7f7d6effd700 1 -- 172.21.15.40:0/4262473729 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7d78024430 con 0x7f7d700052c0 2023-09-30T00:47:58.150 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.7 2023-09-30T00:47:58.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.148+0000 7f7d6effd700 1 --2- 172.21.15.40:0/4262473729 >> v2:172.21.15.135:6800/116426 conn(0x7f7d74068890 0x7f7d7406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:58.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.149+0000 7f7d6effd700 1 -- 172.21.15.40:0/4262473729 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7d780843f0 con 0x7f7d700052c0 2023-09-30T00:47:58.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.149+0000 7f7d6effd700 10 client.4314.objecter ms_dispatch 0x7f7d88001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:58.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.149+0000 7f7d6effd700 3 client.4314.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:58.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.149+0000 7f7d6effd700 3 client.4314.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:58.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.149+0000 7f7d6effd700 20 client.4314.objecter dump_active .. 0 homeless 2023-09-30T00:47:58.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.149+0000 7f7d867fc700 1 --2- 172.21.15.40:0/4262473729 >> v2:172.21.15.135:6800/116426 conn(0x7f7d74068890 0x7f7d7406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:58.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.149+0000 7f7d867fc700 10 monclient: get_auth_request con 0x7f7d74068890 auth_method 0 2023-09-30T00:47:58.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.149+0000 7f7d6effd700 10 client.4314.objecter ms_handle_connect 0x7f7d74068890 2023-09-30T00:47:58.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.149+0000 7f7d867fc700 1 --2- 172.21.15.40:0/4262473729 >> v2:172.21.15.135:6800/116426 conn(0x7f7d74068890 0x7f7d7406ad40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c001620 tx=0x7f7d7c001590 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:58.260 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.259+0000 7fc668f81700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:47:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.259+0000 7fc668f81700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:47:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.260+0000 7fc668f81700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:58.261 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.260+0000 7fc668f81700 1 -- 172.21.15.40:0/2670799539 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc66413ac30 con 0x7fc66413a630 2023-09-30T00:47:58.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.263+0000 7fc6527fc700 1 -- 172.21.15.40:0/2670799539 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc6580554b0 con 0x7fc66413a630 2023-09-30T00:47:58.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.263+0000 7fc6527fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:47:58.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.263+0000 7fc6527fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:58.264 INFO:teuthology.orchestra.run.smithi040.stdout:64424509446 2023-09-30T00:47:58.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.265+0000 7fc668f81700 10 librados: watch_flush enter 2023-09-30T00:47:58.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.265+0000 7fc668f81700 10 librados: watch_flush exit 2023-09-30T00:47:58.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.265+0000 7fc668f81700 20 client.4348.objecter shutdown clearing up homeless session... 2023-09-30T00:47:58.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.265+0000 7fc668f81700 10 client.4348.objecter successfully canceled tick 2023-09-30T00:47:58.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.265+0000 7fc668f81700 1 -- 172.21.15.40:0/2670799539 >> v2:172.21.15.135:6800/116426 conn(0x7fc644068960 msgr2=0x7fc64406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.265+0000 7fc668f81700 1 --2- 172.21.15.40:0/2670799539 >> v2:172.21.15.135:6800/116426 conn(0x7fc644068960 0x7fc64406ae10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc654009890 tx=0x7fc654006d40 comp rx=0 tx=0).stop 2023-09-30T00:47:58.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.265+0000 7fc668f81700 10 monclient: shutdown 2023-09-30T00:47:58.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.265+0000 7fc668f81700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:58.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.265+0000 7fc668f81700 1 -- 172.21.15.40:0/2670799539 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66413a630 msgr2=0x7fc664152100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.265+0000 7fc668f81700 1 --2- 172.21.15.40:0/2670799539 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66413a630 0x7fc664152100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc65800b600 tx=0x7fc6580091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.266+0000 7fc668f81700 1 -- 172.21.15.40:0/2670799539 shutdown_connections 2023-09-30T00:47:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.266+0000 7fc668f81700 1 --2- 172.21.15.40:0/2670799539 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc664152640 0x7fc664138a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.266+0000 7fc668f81700 1 --2- 172.21.15.40:0/2670799539 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64c004f50 0x7fc66413b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.266+0000 7fc668f81700 1 --2- 172.21.15.40:0/2670799539 >> v2:172.21.15.135:6800/116426 conn(0x7fc644068960 0x7fc64406ae10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.266+0000 7fc668f81700 1 --2- 172.21.15.40:0/2670799539 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc66413a630 0x7fc664152100 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.266+0000 7fc668f81700 1 -- 172.21.15.40:0/2670799539 >> 172.21.15.40:0/2670799539 conn(0x7fc66406af40 msgr2=0x7fc664070fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.266+0000 7fc668f81700 1 -- 172.21.15.40:0/2670799539 shutdown_connections 2023-09-30T00:47:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.266+0000 7fc668f81700 1 -- 172.21.15.40:0/2670799539 wait complete. 2023-09-30T00:47:58.268 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.266+0000 7fc668f81700 1 librados: shutdown 2023-09-30T00:47:58.274 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.273+0000 7f8c9cff9700 1 -- 172.21.15.40:0/670833357 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8c9001e820 con 0x7f8ca00b1830 2023-09-30T00:47:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.273+0000 7f8c9cff9700 1 --2- 172.21.15.40:0/670833357 >> v2:172.21.15.135:6800/116426 conn(0x7f8c80068960 0x7f8c8006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.274+0000 7f8c9cff9700 1 -- 172.21.15.40:0/670833357 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8c900862f0 con 0x7f8ca00b1830 2023-09-30T00:47:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.274+0000 7f8c9cff9700 10 client.4283.objecter ms_dispatch 0x7f8ca0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.274+0000 7f8c9cff9700 3 client.4283.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.274+0000 7f8c9cff9700 3 client.4283.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.274+0000 7f8c9cff9700 20 client.4283.objecter dump_active .. 0 homeless 2023-09-30T00:47:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.274+0000 7f8c9ffff700 1 --2- 172.21.15.40:0/670833357 >> v2:172.21.15.135:6800/116426 conn(0x7f8c80068960 0x7f8c8006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:58.275 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.274+0000 7f8c9ffff700 10 monclient: get_auth_request con 0x7f8c80068960 auth_method 0 2023-09-30T00:47:58.276 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.274+0000 7f8c9ffff700 1 --2- 172.21.15.40:0/670833357 >> v2:172.21.15.135:6800/116426 conn(0x7f8c80068960 0x7f8c8006ae10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8c880097c0 tx=0x7f8c88006ce0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:58.276 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.274+0000 7f8c9cff9700 10 client.4283.objecter ms_handle_connect 0x7f8c80068960 2023-09-30T00:47:58.283 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.5 2023-09-30T00:47:58.284 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.284+0000 7f7d6effd700 1 -- 172.21.15.40:0/4262473729 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7d78051e40 con 0x7f7d700052c0 2023-09-30T00:47:58.285 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.284+0000 7f7d6effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:58.285 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.284+0000 7f7d6effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:58.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.294+0000 7f8c9cff9700 1 -- 172.21.15.40:0/670833357 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8c90053d10 con 0x7f8ca00b1830 2023-09-30T00:47:58.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.294+0000 7f8c9cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:58.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.294+0000 7f8c9cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.320+0000 7f8321ffb700 1 -- 172.21.15.40:0/2098221181 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f831c010480 con 0x7f832c0b0f50 2023-09-30T00:47:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.321+0000 7f8321ffb700 1 --2- 172.21.15.40:0/2098221181 >> v2:172.21.15.135:6800/116426 conn(0x7f8310068890 0x7f831006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:58.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.321+0000 7f8331873700 1 --2- 172.21.15.40:0/2098221181 >> v2:172.21.15.135:6800/116426 conn(0x7f8310068890 0x7f831006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:58.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.321+0000 7f8331873700 10 monclient: get_auth_request con 0x7f8310068890 auth_method 0 2023-09-30T00:47:58.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.322+0000 7f8321ffb700 10 client.4317.objecter ms_handle_connect 0x7f8310068890 2023-09-30T00:47:58.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.322+0000 7f8331873700 1 --2- 172.21.15.40:0/2098221181 >> v2:172.21.15.135:6800/116426 conn(0x7f8310068890 0x7f831006ad40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8324009fd0 tx=0x7f8324009620 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:58.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.329+0000 7f1b35ffb700 1 -- 172.21.15.40:0/846249853 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1b30057160 con 0x7f1b400a7610 2023-09-30T00:47:58.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.329+0000 7f1b35ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:47:58.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.329+0000 7f1b35ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:58.330 INFO:teuthology.orchestra.run.smithi040.stdout:85899345925 2023-09-30T00:47:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.332+0000 7f1b481c1700 10 librados: watch_flush enter 2023-09-30T00:47:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.332+0000 7f1b481c1700 10 librados: watch_flush exit 2023-09-30T00:47:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.332+0000 7f1b481c1700 20 client.4302.objecter shutdown clearing up homeless session... 2023-09-30T00:47:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.332+0000 7f1b481c1700 10 client.4302.objecter successfully canceled tick 2023-09-30T00:47:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.332+0000 7f1b481c1700 1 -- 172.21.15.40:0/846249853 >> v2:172.21.15.135:6800/116426 conn(0x7f1b240689e0 msgr2=0x7f1b2406ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.332+0000 7f1b481c1700 1 --2- 172.21.15.40:0/846249853 >> v2:172.21.15.135:6800/116426 conn(0x7f1b240689e0 0x7f1b2406ae90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c006fd0 tx=0x7f1b3c006db0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.332+0000 7f1b481c1700 10 monclient: shutdown 2023-09-30T00:47:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.332+0000 7f1b481c1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.332+0000 7f1b481c1700 1 -- 172.21.15.40:0/846249853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b400a7610 msgr2=0x7f1b400a8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.332+0000 7f1b481c1700 1 --2- 172.21.15.40:0/846249853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b400a7610 0x7f1b400a8620 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1b300091e0 tx=0x7f1b30016000 comp rx=0 tx=0).stop 2023-09-30T00:47:58.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.333+0000 7f1b481c1700 1 -- 172.21.15.40:0/846249853 shutdown_connections 2023-09-30T00:47:58.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.333+0000 7f1b481c1700 1 --2- 172.21.15.40:0/846249853 >> v2:172.21.15.135:6800/116426 conn(0x7f1b240689e0 0x7f1b2406ae90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.333+0000 7f1b481c1700 1 --2- 172.21.15.40:0/846249853 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1b400b16c0 0x7f1b400a9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.333+0000 7f1b481c1700 1 --2- 172.21.15.40:0/846249853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b400a7610 0x7f1b400a8620 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.333+0000 7f1b481c1700 1 --2- 172.21.15.40:0/846249853 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1b400a7f80 0x7f1b4013ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.333+0000 7f1b481c1700 1 -- 172.21.15.40:0/846249853 >> 172.21.15.40:0/846249853 conn(0x7f1b4006f130 msgr2=0x7f1b400b2890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:58.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.333+0000 7f1b481c1700 1 -- 172.21.15.40:0/846249853 shutdown_connections 2023-09-30T00:47:58.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.333+0000 7f1b481c1700 1 -- 172.21.15.40:0/846249853 wait complete. 2023-09-30T00:47:58.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.333+0000 7f1b481c1700 1 librados: shutdown 2023-09-30T00:47:58.352 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345927 got 85899345925 for osd.3 2023-09-30T00:47:58.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.359+0000 7f8321ffb700 1 -- 172.21.15.40:0/2098221181 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f831c019ea0 con 0x7f832c0b0f50 2023-09-30T00:47:58.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.359+0000 7f8321ffb700 10 client.4317.objecter ms_dispatch 0x7f832c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:58.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.359+0000 7f8321ffb700 3 client.4317.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:58.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.359+0000 7f8321ffb700 3 client.4317.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:58.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.359+0000 7f8321ffb700 20 client.4317.objecter dump_active .. 0 homeless 2023-09-30T00:47:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.509+0000 7f8ca65cb700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:47:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.509+0000 7f8ca65cb700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:47:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.509+0000 7f8ca65cb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:58.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.509+0000 7f8ca65cb700 1 -- 172.21.15.40:0/670833357 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8c84005630 con 0x7f8ca00b1830 2023-09-30T00:47:58.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.512+0000 7f7d8d530700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T00:47:58.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.512+0000 7f7d8d530700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T00:47:58.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.512+0000 7f7d8d530700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:58.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.512+0000 7f7d8d530700 1 -- 172.21.15.40:0/4262473729 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f7d8806c890 con 0x7f7d700052c0 2023-09-30T00:47:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.593+0000 7f74f3fff700 1 -- 172.21.15.40:0/229920520 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7500053940 con 0x7f750c06dd20 2023-09-30T00:47:58.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.593+0000 7f74f3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.593+0000 7f74f3fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:58.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.605+0000 7f8c9cff9700 1 -- 172.21.15.40:0/670833357 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8c90057330 con 0x7f8ca00b1830 2023-09-30T00:47:58.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.606+0000 7f8c9cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:47:58.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.606+0000 7f8c9cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:58.607 INFO:teuthology.orchestra.run.smithi040.stdout:81604378629 2023-09-30T00:47:58.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.606+0000 7f96877fe700 1 -- 172.21.15.40:0/4290983236 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9694055390 con 0x7f96a006beb0 2023-09-30T00:47:58.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.606+0000 7f96877fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:47:58.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.606+0000 7f96877fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:58.608 INFO:teuthology.orchestra.run.smithi040.stdout:85899345925 2023-09-30T00:47:58.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 10 librados: watch_flush enter 2023-09-30T00:47:58.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 10 librados: watch_flush exit 2023-09-30T00:47:58.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 20 client.4283.objecter shutdown clearing up homeless session... 2023-09-30T00:47:58.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 10 client.4283.objecter successfully canceled tick 2023-09-30T00:47:58.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 1 -- 172.21.15.40:0/670833357 >> v2:172.21.15.135:6800/116426 conn(0x7f8c80068960 msgr2=0x7f8c8006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 1 --2- 172.21.15.40:0/670833357 >> v2:172.21.15.135:6800/116426 conn(0x7f8c80068960 0x7f8c8006ae10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8c880097c0 tx=0x7f8c88006ce0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 10 monclient: shutdown 2023-09-30T00:47:58.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:58.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 1 -- 172.21.15.40:0/670833357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ca00b1830 msgr2=0x7f8ca00a8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 1 --2- 172.21.15.40:0/670833357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ca00b1830 0x7f8ca00a8640 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000b6c0 tx=0x7f8c90017000 comp rx=0 tx=0).stop 2023-09-30T00:47:58.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 1 -- 172.21.15.40:0/670833357 shutdown_connections 2023-09-30T00:47:58.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 1 --2- 172.21.15.40:0/670833357 >> v2:172.21.15.135:6800/116426 conn(0x7f8c80068960 0x7f8c8006ae10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 1 --2- 172.21.15.40:0/670833357 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8ca00b1460 0x7f8ca00a93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 1 --2- 172.21.15.40:0/670833357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ca013ac20 0x7f8ca013d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 1 --2- 172.21.15.40:0/670833357 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8ca00b1830 0x7f8ca00a8640 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.608+0000 7f8ca65cb700 1 -- 172.21.15.40:0/670833357 >> 172.21.15.40:0/670833357 conn(0x7f8ca00631f0 msgr2=0x7f8ca0142870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:58.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.609+0000 7f96a61d1700 10 librados: watch_flush enter 2023-09-30T00:47:58.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.609+0000 7f96a61d1700 10 librados: watch_flush exit 2023-09-30T00:47:58.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.609+0000 7f96a61d1700 20 client.4274.objecter shutdown clearing up homeless session... 2023-09-30T00:47:58.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.609+0000 7f96a61d1700 10 client.4274.objecter successfully canceled tick 2023-09-30T00:47:58.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.609+0000 7f96a61d1700 1 -- 172.21.15.40:0/4290983236 >> v2:172.21.15.135:6800/116426 conn(0x7f968c0688c0 msgr2=0x7f968c06ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.609+0000 7f96a61d1700 1 --2- 172.21.15.40:0/4290983236 >> v2:172.21.15.135:6800/116426 conn(0x7f968c0688c0 0x7f968c06ad70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9690006fd0 tx=0x7f9690006db0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.609+0000 7f96a61d1700 10 monclient: shutdown 2023-09-30T00:47:58.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.609+0000 7f96a61d1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:58.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.609+0000 7f96a61d1700 1 -- 172.21.15.40:0/4290983236 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96a006beb0 msgr2=0x7f96a0148160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.609+0000 7f96a61d1700 1 --2- 172.21.15.40:0/4290983236 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96a006beb0 0x7f96a0148160 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f969400b7e0 tx=0x7f96940091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.609+0000 7f8ca65cb700 1 -- 172.21.15.40:0/670833357 shutdown_connections 2023-09-30T00:47:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.610+0000 7f96a61d1700 1 -- 172.21.15.40:0/4290983236 shutdown_connections 2023-09-30T00:47:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.610+0000 7f96a61d1700 1 --2- 172.21.15.40:0/4290983236 >> v2:172.21.15.135:6800/116426 conn(0x7f968c0688c0 0x7f968c06ad70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.610+0000 7f96a61d1700 1 --2- 172.21.15.40:0/4290983236 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96a01432a0 0x7f96a00ba640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.610+0000 7f96a61d1700 1 --2- 172.21.15.40:0/4290983236 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96a006ad50 0x7f96a00b8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.610+0000 7f96a61d1700 1 --2- 172.21.15.40:0/4290983236 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96a006beb0 0x7f96a0148160 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.610+0000 7f96a61d1700 1 -- 172.21.15.40:0/4290983236 >> 172.21.15.40:0/4290983236 conn(0x7f96a00b4aa0 msgr2=0x7f96a00b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.610+0000 7f8ca65cb700 1 -- 172.21.15.40:0/670833357 wait complete. 2023-09-30T00:47:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.610+0000 7f8ca65cb700 1 librados: shutdown 2023-09-30T00:47:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.610+0000 7f96a61d1700 1 -- 172.21.15.40:0/4290983236 shutdown_connections 2023-09-30T00:47:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.610+0000 7f96a61d1700 1 -- 172.21.15.40:0/4290983236 wait complete. 2023-09-30T00:47:58.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.610+0000 7f96a61d1700 1 librados: shutdown 2023-09-30T00:47:58.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345927 got 85899345925 for osd.0 2023-09-30T00:47:58.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378630 got 81604378629 for osd.1 2023-09-30T00:47:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.721+0000 7f8321ffb700 1 -- 172.21.15.40:0/2098221181 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f831c052040 con 0x7f832c0b0f50 2023-09-30T00:47:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.722+0000 7f8321ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.722+0000 7f8321ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.723+0000 7f7d6effd700 1 -- 172.21.15.40:0/4262473729 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7d78055460 con 0x7f7d700052c0 2023-09-30T00:47:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.723+0000 7f7d6effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T00:47:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.723+0000 7f7d6effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:58.724 INFO:teuthology.orchestra.run.smithi040.stdout:60129542150 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 10 librados: watch_flush enter 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 10 librados: watch_flush exit 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 20 client.4314.objecter shutdown clearing up homeless session... 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 10 client.4314.objecter successfully canceled tick 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 1 -- 172.21.15.40:0/4262473729 >> v2:172.21.15.135:6800/116426 conn(0x7f7d74068890 msgr2=0x7f7d7406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 1 --2- 172.21.15.40:0/4262473729 >> v2:172.21.15.135:6800/116426 conn(0x7f7d74068890 0x7f7d7406ad40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c001620 tx=0x7f7d7c001590 comp rx=0 tx=0).stop 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 10 monclient: shutdown 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 1 -- 172.21.15.40:0/4262473729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d700052c0 msgr2=0x7f7d8806d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 1 --2- 172.21.15.40:0/4262473729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d700052c0 0x7f7d8806d5b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7d7800b580 tx=0x7f7d780091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 1 -- 172.21.15.40:0/4262473729 shutdown_connections 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 1 --2- 172.21.15.40:0/4262473729 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7d8806ae80 0x7f7d8814cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 1 --2- 172.21.15.40:0/4262473729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d700052c0 0x7f7d8806d5b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 1 --2- 172.21.15.40:0/4262473729 >> v2:172.21.15.135:6800/116426 conn(0x7f7d74068890 0x7f7d7406ad40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 1 --2- 172.21.15.40:0/4262473729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7d8806dc10 0x7f7d8806a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.726+0000 7f7d8d530700 1 -- 172.21.15.40:0/4262473729 >> 172.21.15.40:0/4262473729 conn(0x7f7d880631f0 msgr2=0x7f7d880af310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.727+0000 7f7d8d530700 1 -- 172.21.15.40:0/4262473729 shutdown_connections 2023-09-30T00:47:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.727+0000 7f7d8d530700 1 -- 172.21.15.40:0/4262473729 wait complete. 2023-09-30T00:47:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.727+0000 7f7d8d530700 1 librados: shutdown 2023-09-30T00:47:58.742 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.6 2023-09-30T00:47:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.831+0000 7f7512867700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:47:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.831+0000 7f7512867700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:47:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.831+0000 7f7512867700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.831+0000 7f7512867700 1 -- 172.21.15.40:0/229920520 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f750c1554d0 con 0x7f750c06dd20 2023-09-30T00:47:58.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.837+0000 7f74f3fff700 1 -- 172.21.15.40:0/229920520 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7500056f60 con 0x7f750c06dd20 2023-09-30T00:47:58.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.837+0000 7f74f3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:47:58.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.837+0000 7f74f3fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:58.838 INFO:teuthology.orchestra.run.smithi040.stdout:64424509445 2023-09-30T00:47:58.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 10 librados: watch_flush enter 2023-09-30T00:47:58.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 10 librados: watch_flush exit 2023-09-30T00:47:58.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 20 client.4292.objecter shutdown clearing up homeless session... 2023-09-30T00:47:58.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 10 client.4292.objecter successfully canceled tick 2023-09-30T00:47:58.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 1 -- 172.21.15.40:0/229920520 >> v2:172.21.15.135:6800/116426 conn(0x7f74f8068890 msgr2=0x7f74f806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 1 --2- 172.21.15.40:0/229920520 >> v2:172.21.15.135:6800/116426 conn(0x7f74f8068890 0x7f74f806ad40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f750c06d5f0 tx=0x7f74fc006c40 comp rx=0 tx=0).stop 2023-09-30T00:47:58.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 10 monclient: shutdown 2023-09-30T00:47:58.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:58.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 1 -- 172.21.15.40:0/229920520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f750c06dd20 msgr2=0x7f750c06c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 1 --2- 172.21.15.40:0/229920520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f750c06dd20 0x7f750c06c520 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7500009930 tx=0x7f7500016000 comp rx=0 tx=0).stop 2023-09-30T00:47:58.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 1 -- 172.21.15.40:0/229920520 shutdown_connections 2023-09-30T00:47:58.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 1 --2- 172.21.15.40:0/229920520 >> v2:172.21.15.135:6800/116426 conn(0x7f74f8068890 0x7f74f806ad40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 1 --2- 172.21.15.40:0/229920520 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f750c0b1870 0x7f750c0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 1 --2- 172.21.15.40:0/229920520 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f750c06dd20 0x7f750c06c520 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 1 --2- 172.21.15.40:0/229920520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f750c0b14a0 0x7f750c0b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 1 -- 172.21.15.40:0/229920520 >> 172.21.15.40:0/229920520 conn(0x7f750c0631f0 msgr2=0x7f750c145060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:58.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 1 -- 172.21.15.40:0/229920520 shutdown_connections 2023-09-30T00:47:58.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 1 -- 172.21.15.40:0/229920520 wait complete. 2023-09-30T00:47:58.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.841+0000 7f7512867700 1 librados: shutdown 2023-09-30T00:47:58.858 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.4 2023-09-30T00:47:58.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.953+0000 7f8333ad7700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:47:58.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.953+0000 7f8333ad7700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:47:58.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.953+0000 7f8333ad7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:58.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.953+0000 7f8333ad7700 1 -- 172.21.15.40:0/2098221181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f830c005630 con 0x7f832c0b0f50 2023-09-30T00:47:58.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.960+0000 7f8321ffb700 1 -- 172.21.15.40:0/2098221181 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f831c059e60 con 0x7f832c0b0f50 2023-09-30T00:47:58.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.960+0000 7f8321ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:47:58.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.960+0000 7f8321ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:58.961 INFO:teuthology.orchestra.run.smithi040.stdout:85899345924 2023-09-30T00:47:58.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 10 librados: watch_flush enter 2023-09-30T00:47:58.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 10 librados: watch_flush exit 2023-09-30T00:47:58.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 20 client.4317.objecter shutdown clearing up homeless session... 2023-09-30T00:47:58.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 10 client.4317.objecter successfully canceled tick 2023-09-30T00:47:58.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 1 -- 172.21.15.40:0/2098221181 >> v2:172.21.15.135:6800/116426 conn(0x7f8310068890 msgr2=0x7f831006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 1 --2- 172.21.15.40:0/2098221181 >> v2:172.21.15.135:6800/116426 conn(0x7f8310068890 0x7f831006ad40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8324009fd0 tx=0x7f8324009620 comp rx=0 tx=0).stop 2023-09-30T00:47:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 10 monclient: shutdown 2023-09-30T00:47:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 1 -- 172.21.15.40:0/2098221181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f832c0b0f50 msgr2=0x7f832c0a9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 1 --2- 172.21.15.40:0/2098221181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f832c0b0f50 0x7f832c0a9c60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f831c00b810 tx=0x7f831c009250 comp rx=0 tx=0).stop 2023-09-30T00:47:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 1 -- 172.21.15.40:0/2098221181 shutdown_connections 2023-09-30T00:47:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 1 --2- 172.21.15.40:0/2098221181 >> v2:172.21.15.135:6800/116426 conn(0x7f8310068890 0x7f831006ad40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 1 --2- 172.21.15.40:0/2098221181 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f832c145d30 0x7f832c12f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 1 --2- 172.21.15.40:0/2098221181 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f832c06f0f0 0x7f832c147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 1 --2- 172.21.15.40:0/2098221181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f832c0b0f50 0x7f832c0a9c60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 1 -- 172.21.15.40:0/2098221181 >> 172.21.15.40:0/2098221181 conn(0x7f832c067bf0 msgr2=0x7f832c0ad810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 1 -- 172.21.15.40:0/2098221181 shutdown_connections 2023-09-30T00:47:58.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 1 -- 172.21.15.40:0/2098221181 wait complete. 2023-09-30T00:47:58.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:58.963+0000 7f8333ad7700 1 librados: shutdown 2023-09-30T00:47:58.977 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345926 got 85899345924 for osd.2 2023-09-30T00:47:59.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T00:47:59.284 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T00:47:59.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.290+0000 7f530fa15700 10 monclient: get_monmap_and_config 2023-09-30T00:47:59.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.290+0000 7f530fa15700 10 monclient: build_initial_monmap 2023-09-30T00:47:59.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.290+0000 7f530fa15700 10 monclient: monmap: 2023-09-30T00:47:59.291 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:59.291 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.291 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:59.291187+0000 2023-09-30T00:47:59.291 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:59.291187+0000 2023-09-30T00:47:59.291 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:59.291 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.291 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:59.291 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:59.292 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:59.292 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:59.292 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:59.292 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.293 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.290+0000 7f530fa15700 1 Processor -- start 2023-09-30T00:47:59.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.291+0000 7f530fa15700 1 -- start start 2023-09-30T00:47:59.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.291+0000 7f530fa15700 10 monclient: init 2023-09-30T00:47:59.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.291+0000 7f530fa15700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:59.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.291+0000 7f530fa15700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-09-30T00:47:59.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.291+0000 7f530fa15700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f53080ae830 0x7f53080acfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.291+0000 7f530fa15700 10 monclient(hunting): picked mon.noname-c con 0x7f53080ae830 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:59.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.291+0000 7f530fa15700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53080ad4e0 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:59.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.291+0000 7f530fa15700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5308067d30 0x7f5308148e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.291+0000 7f530fa15700 10 monclient(hunting): picked mon.noname-b con 0x7f5308067d30 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:59.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.291+0000 7f530fa15700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530806fc90 con 0x7f53080ad4e0 2023-09-30T00:47:59.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.292+0000 7f530fa15700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.292+0000 7f530fa15700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53080a60f0 con 0x7f53080ae830 2023-09-30T00:47:59.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.292+0000 7f530fa15700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.292+0000 7f530fa15700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53080aec00 con 0x7f5308067d30 2023-09-30T00:47:59.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.292+0000 7f530fa15700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:59.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.292+0000 7f530fa15700 10 monclient(hunting): authenticate will time out at 5340.593027s 2023-09-30T00:47:59.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.293+0000 7f530dfb2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5308067d30 0x7f5308148e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.293+0000 7f530dfb2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5308067d30 0x7f5308148e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36688/0 (socket says 172.21.15.40:36688) 2023-09-30T00:47:59.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.293+0000 7f530dfb2700 1 -- 172.21.15.40:0/1527631344 learned_addr learned my addr 172.21.15.40:0/1527631344 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:59.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.293+0000 7f530dfb2700 10 monclient(hunting): get_auth_request con 0x7f5308067d30 auth_method 0 2023-09-30T00:47:59.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.293+0000 7f530dfb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:59.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.293+0000 7f530dfb2700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.293+0000 7f530dfb2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:59.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.293+0000 7f530d7b1700 1 --2- 172.21.15.40:0/1527631344 >> v2:172.21.15.135:3300/0 conn(0x7f53080ae830 0x7f53080acfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.294+0000 7f530dfb2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:59.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.294+0000 7f530dfb2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:59.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.294+0000 7f530dfb2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:59.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.294+0000 7f530d7b1700 10 monclient(hunting): get_auth_request con 0x7f53080ae830 auth_method 0 2023-09-30T00:47:59.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.294+0000 7f530d7b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:59.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.294+0000 7f530d7b1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.294+0000 7f530d7b1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:59.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.296+0000 7f530dfb2700 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2023-09-30T00:47:59.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.296+0000 7f530dfb2700 1 -- 172.21.15.40:0/1527631344 >> v2:172.21.15.135:3300/0 conn(0x7f53080ae830 msgr2=0x7f53080acfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:59.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.296+0000 7f530dfb2700 1 --2- 172.21.15.40:0/1527631344 >> v2:172.21.15.135:3300/0 conn(0x7f53080ae830 0x7f53080acfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.296+0000 7f530dfb2700 1 -- 172.21.15.40:0/1527631344 >> v1:172.21.15.135:6789/0 conn(0x7f53080ad4e0 legacy=0x7f5308067620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.296+0000 7f530dfb2700 10 monclient: _finish_hunting 0 2023-09-30T00:47:59.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.296+0000 7f530dfb2700 1 monclient: found mon.noname-b 2023-09-30T00:47:59.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.296+0000 7f530dfb2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:59.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.296+0000 7f530dfb2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:47:59.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.296+0000 7f530dfb2700 1 -- 172.21.15.40:0/1527631344 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53080ad960 con 0x7f5308067d30 2023-09-30T00:47:59.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.300+0000 7f530dfb2700 1 --2- 172.21.15.40:0/1527631344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5308067d30 0x7f5308148e90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5300009f20 tx=0x7f5300009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9f9b1a468174298 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:59.306 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.305+0000 7f52fffff700 1 -- 172.21.15.40:0/1527631344 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530000e070 con 0x7f5308067d30 2023-09-30T00:47:59.306 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.305+0000 7f52fffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.306 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.305+0000 7f52fffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:59.306 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.305+0000 7f52fffff700 10 monclient: dump: 2023-09-30T00:47:59.306 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.306 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.306 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.307 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.305+0000 7f52fffff700 10 monclient: _finish_auth 0 2023-09-30T00:47:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.305+0000 7f52fffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:59.307 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.305+0000 7f530fa15700 5 monclient: authenticate success, global_id 4295 2023-09-30T00:47:59.308 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.305+0000 7f530fa15700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:59.308 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.307+0000 7f52fffff700 1 -- 172.21.15.40:0/1527631344 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530000ba80 con 0x7f5308067d30 2023-09-30T00:47:59.308 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.307+0000 7f52fffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:59.308 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.307+0000 7f52fffff700 1 -- 172.21.15.40:0/1527631344 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53000092e0 con 0x7f5308067d30 2023-09-30T00:47:59.308 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.307+0000 7f530fa15700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:59.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.307+0000 7f530fa15700 4 set_mon_vals no callback set 2023-09-30T00:47:59.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.307+0000 7f52fffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.307+0000 7f52fffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:59.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.307+0000 7f52fffff700 10 monclient: dump: 2023-09-30T00:47:59.309 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.309 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.309 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.309 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.309 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.309 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.310 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.310 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.310 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.310 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.308+0000 7f530fa15700 10 monclient: shutdown 2023-09-30T00:47:59.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.308+0000 7f530fa15700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.308+0000 7f530fa15700 1 -- 172.21.15.40:0/1527631344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5308067d30 msgr2=0x7f5308148e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.308+0000 7f530fa15700 1 --2- 172.21.15.40:0/1527631344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5308067d30 0x7f5308148e90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5300009f20 tx=0x7f5300009840 comp rx=0 tx=0).stop 2023-09-30T00:47:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.308+0000 7f530fa15700 1 -- 172.21.15.40:0/1527631344 shutdown_connections 2023-09-30T00:47:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.308+0000 7f530fa15700 1 --2- 172.21.15.40:0/1527631344 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5308067d30 0x7f5308148e90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.308+0000 7f530fa15700 1 --2- 172.21.15.40:0/1527631344 >> v2:172.21.15.135:3300/0 conn(0x7f53080ae830 0x7f53080acfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.308+0000 7f530fa15700 1 -- 172.21.15.40:0/1527631344 >> 172.21.15.40:0/1527631344 conn(0x7f5308068920 msgr2=0x7f5308068d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.308+0000 7f530fa15700 1 -- 172.21.15.40:0/1527631344 shutdown_connections 2023-09-30T00:47:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.308+0000 7f530fa15700 1 -- 172.21.15.40:0/1527631344 wait complete. 2023-09-30T00:47:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.309+0000 7f530fa15700 10 monclient: build_initial_monmap 2023-09-30T00:47:59.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.309+0000 7f530fa15700 10 monclient: monmap: 2023-09-30T00:47:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:59.315 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.310+0000 7f530fa15700 1 librados: starting msgr at 2023-09-30T00:47:59.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.310+0000 7f530fa15700 1 librados: starting objecter 2023-09-30T00:47:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.310+0000 7f530fa15700 1 Processor -- start 2023-09-30T00:47:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.310+0000 7f530fa15700 1 -- start start 2023-09-30T00:47:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.310+0000 7f530fa15700 1 librados: setting wanted keys 2023-09-30T00:47:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.310+0000 7f530fa15700 1 librados: calling monclient init 2023-09-30T00:47:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.310+0000 7f530fa15700 10 monclient: init 2023-09-30T00:47:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:47:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5308067d30 0x7f5308122d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 10 monclient(hunting): picked mon.noname-a con 0x7f5308067d30 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53080ad4e0 0x7f5308121fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 10 monclient(hunting): picked mon.noname-c con 0x7f53080ad4e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:59.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53080ae830 0x7f530811f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 10 monclient(hunting): picked mon.noname-b con 0x7f53080ae830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5308123350 con 0x7f5308067d30 2023-09-30T00:47:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5308124150 con 0x7f53080ae830 2023-09-30T00:47:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53081243c0 con 0x7f53080ad4e0 2023-09-30T00:47:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.311+0000 7f530fa15700 10 monclient(hunting): authenticate will time out at 5340.612574s 2023-09-30T00:47:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.312+0000 7f530d7b1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5308067d30 0x7f5308122d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.312+0000 7f530dfb2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53080ae830 0x7f530811f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530d7b1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5308067d30 0x7f5308122d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56700/0 (socket says 172.21.15.40:56700) 2023-09-30T00:47:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530d7b1700 1 -- 172.21.15.40:0/788429292 learned_addr learned my addr 172.21.15.40:0/788429292 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530d7b1700 10 monclient(hunting): get_auth_request con 0x7f5308067d30 auth_method 0 2023-09-30T00:47:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530d7b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530d7b1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530d7b1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530dfb2700 10 monclient(hunting): get_auth_request con 0x7f53080ae830 auth_method 0 2023-09-30T00:47:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530dfb2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530dfb2700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530dfb2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:59.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530dfb2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:59.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530dfb2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:59.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530dfb2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:59.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530d7b1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:59.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530d7b1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:59.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.313+0000 7f530d7b1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:59.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.314+0000 7f530cfb0700 1 --2- 172.21.15.40:0/788429292 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53080ad4e0 0x7f5308121fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.315+0000 7f530cfb0700 10 monclient(hunting): get_auth_request con 0x7f53080ad4e0 auth_method 0 2023-09-30T00:47:59.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.315+0000 7f530cfb0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:59.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.315+0000 7f530cfb0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.315+0000 7f530cfb0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:59.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.315+0000 7f530cfb0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:59.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.315+0000 7f530cfb0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:59.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.315+0000 7f530cfb0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:59.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.317+0000 7f530dfb2700 10 monclient(hunting): handle_auth_done global_id 4357 payload 824 2023-09-30T00:47:59.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.317+0000 7f530dfb2700 1 -- 172.21.15.40:0/788429292 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53080ad4e0 msgr2=0x7f5308121fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.317+0000 7f530dfb2700 1 --2- 172.21.15.40:0/788429292 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53080ad4e0 0x7f5308121fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.317+0000 7f530dfb2700 1 -- 172.21.15.40:0/788429292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5308067d30 msgr2=0x7f5308122d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.317+0000 7f530dfb2700 1 --2- 172.21.15.40:0/788429292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5308067d30 0x7f5308122d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.317+0000 7f530dfb2700 10 monclient: _finish_hunting 0 2023-09-30T00:47:59.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.317+0000 7f530dfb2700 1 monclient: found mon.noname-b 2023-09-30T00:47:59.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.317+0000 7f530dfb2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:59.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.317+0000 7f530dfb2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:47:59.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.317+0000 7f530dfb2700 1 -- 172.21.15.40:0/788429292 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5300009ae0 con 0x7f53080ae830 2023-09-30T00:47:59.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.320+0000 7f530dfb2700 1 --2- 172.21.15.40:0/788429292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53080ae830 0x7f530811f650 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5300015000 tx=0x7f5300018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:59.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.320+0000 7f52fd7fa700 10 client.?.objecter ms_handle_connect 0x7f53080ae830 2023-09-30T00:47:59.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.320+0000 7f52fd7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:59.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f52fd7fa700 1 -- 172.21.15.40:0/788429292 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53000094e0 con 0x7f53080ae830 2023-09-30T00:47:59.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f52fd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f52fd7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f52fd7fa700 10 monclient: dump: 2023-09-30T00:47:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.324 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.325 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.325 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.325 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f52fd7fa700 10 monclient: _finish_auth 0 2023-09-30T00:47:59.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f52fd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:59.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f530fa15700 5 monclient: authenticate success, global_id 4357 2023-09-30T00:47:59.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f530fa15700 10 monclient: _renew_subs 2023-09-30T00:47:59.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f530fa15700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:59.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f530fa15700 1 -- 172.21.15.40:0/788429292 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5308121120 con 0x7f53080ae830 2023-09-30T00:47:59.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f530fa15700 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:59.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f530fa15700 10 monclient: _renew_subs 2023-09-30T00:47:59.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f530fa15700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:59.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f530fa15700 1 -- 172.21.15.40:0/788429292 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530811fe30 con 0x7f53080ae830 2023-09-30T00:47:59.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.322+0000 7f530fa15700 1 librados: init done 2023-09-30T00:47:59.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.323+0000 7f530fa15700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:59.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.323+0000 7f530fa15700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:59.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.323+0000 7f530fa15700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:59.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.323+0000 7f530fa15700 1 -- 172.21.15.40:0/788429292 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5308064a00 con 0x7f53080ae830 2023-09-30T00:47:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.344+0000 7f52fd7fa700 1 -- 172.21.15.40:0/788429292 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5300007170 con 0x7f53080ae830 2023-09-30T00:47:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.344+0000 7f52fd7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.344+0000 7f52fd7fa700 1 -- 172.21.15.40:0/788429292 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53000032f0 con 0x7f53080ae830 2023-09-30T00:47:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.344+0000 7f52fd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.344+0000 7f52fd7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.344+0000 7f52fd7fa700 10 monclient: dump: 2023-09-30T00:47:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.345 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.346 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.344+0000 7f52ff7fe700 4 set_mon_vals no callback set 2023-09-30T00:47:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.345+0000 7f52fd7fa700 1 -- 172.21.15.40:0/788429292 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5300003040 con 0x7f53080ae830 2023-09-30T00:47:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.345+0000 7f52fd7fa700 1 --2- 172.21.15.40:0/788429292 >> v2:172.21.15.135:6800/116426 conn(0x7f52ec068890 0x7f52ec06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.345+0000 7f530d7b1700 1 --2- 172.21.15.40:0/788429292 >> v2:172.21.15.135:6800/116426 conn(0x7f52ec068890 0x7f52ec06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.345+0000 7f530d7b1700 10 monclient: get_auth_request con 0x7f52ec068890 auth_method 0 2023-09-30T00:47:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.346+0000 7f530d7b1700 1 --2- 172.21.15.40:0/788429292 >> v2:172.21.15.135:6800/116426 conn(0x7f52ec068890 0x7f52ec06ad40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5304006fd0 tx=0x7f5304008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:59.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.346+0000 7f52fd7fa700 10 client.4357.objecter ms_handle_connect 0x7f52ec068890 2023-09-30T00:47:59.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:47:59.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.431+0000 7f1d7e6e7700 10 monclient: get_monmap_and_config 2023-09-30T00:47:59.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.431+0000 7f1d7e6e7700 10 monclient: build_initial_monmap 2023-09-30T00:47:59.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.431+0000 7f1d7e6e7700 10 monclient: monmap: 2023-09-30T00:47:59.432 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:59.432 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.432 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:59.432179+0000 2023-09-30T00:47:59.432 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:59.432179+0000 2023-09-30T00:47:59.432 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:59.432 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.432 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:59.433 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:59.433 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:59.433 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:59.433 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:59.433 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.431+0000 7f1d7e6e7700 1 Processor -- start 2023-09-30T00:47:59.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 1 -- start start 2023-09-30T00:47:59.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 10 monclient: init 2023-09-30T00:47:59.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:59.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-09-30T00:47:59.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1d780b1460 0x7f1d780b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 10 monclient(hunting): picked mon.noname-a con 0x7f1d780b1460 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:59.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1d780a5fb0 0x7f1d78141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 10 monclient(hunting): picked mon.noname-c con 0x7f1d780a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:59.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d780af410 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:59.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d780a71f0 con 0x7f1d780af410 2023-09-30T00:47:59.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d780a74d0 con 0x7f1d780b1460 2023-09-30T00:47:59.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d780a63c0 con 0x7f1d780a5fb0 2023-09-30T00:47:59.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:59.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.432+0000 7f1d7e6e7700 10 monclient(hunting): authenticate will time out at 5340.733816s 2023-09-30T00:47:59.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.433+0000 7f1d77fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1d780b1460 0x7f1d780b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.433+0000 7f1d77fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1d780b1460 0x7f1d780b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56706/0 (socket says 172.21.15.40:56706) 2023-09-30T00:47:59.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.433+0000 7f1d77fff700 1 -- 172.21.15.40:0/912674331 learned_addr learned my addr 172.21.15.40:0/912674331 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:59.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.433+0000 7f1d77fff700 10 monclient(hunting): get_auth_request con 0x7f1d780b1460 auth_method 0 2023-09-30T00:47:59.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.433+0000 7f1d77fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:59.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.433+0000 7f1d77fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.433+0000 7f1d77fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:59.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.434+0000 7f1d77fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:59.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.434+0000 7f1d77fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:59.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.434+0000 7f1d77fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:59.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.436+0000 7f1d77fff700 10 monclient(hunting): handle_auth_done global_id 4323 payload 274 2023-09-30T00:47:59.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.436+0000 7f1d77fff700 1 -- 172.21.15.40:0/912674331 >> v2:172.21.15.135:3300/0 conn(0x7f1d780a5fb0 msgr2=0x7f1d78141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:59.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.436+0000 7f1d77fff700 1 --2- 172.21.15.40:0/912674331 >> v2:172.21.15.135:3300/0 conn(0x7f1d780a5fb0 0x7f1d78141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.436+0000 7f1d77fff700 1 -- 172.21.15.40:0/912674331 >> v1:172.21.15.40:6789/0 conn(0x7f1d780af410 legacy=0x7f1d780af870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:59.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.436+0000 7f1d77fff700 10 monclient: _finish_hunting 0 2023-09-30T00:47:59.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.436+0000 7f1d77fff700 1 monclient: found mon.noname-a 2023-09-30T00:47:59.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.436+0000 7f1d77fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:59.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.436+0000 7f1d77fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:59.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.436+0000 7f1d77fff700 1 -- 172.21.15.40:0/912674331 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d780a56d0 con 0x7f1d780b1460 2023-09-30T00:47:59.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.439+0000 7f1d77fff700 1 --2- 172.21.15.40:0/912674331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d780b1460 0x7f1d780b1830 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1d680099b0 tx=0x7f1d68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9dcf509eb465271 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:59.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d76ffd700 1 -- 172.21.15.40:0/912674331 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6800b7e0 con 0x7f1d780b1460 2023-09-30T00:47:59.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d76ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d76ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:59.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d76ffd700 10 monclient: dump: 2023-09-30T00:47:59.446 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.447 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.447 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.447 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.447 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.447 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.447 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.447 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.447 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.447 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d76ffd700 10 monclient: _finish_auth 0 2023-09-30T00:47:59.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d76ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:59.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d7e6e7700 5 monclient: authenticate success, global_id 4323 2023-09-30T00:47:59.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d7e6e7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:59.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d76ffd700 1 -- 172.21.15.40:0/912674331 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6800b940 con 0x7f1d780b1460 2023-09-30T00:47:59.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d76ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:59.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d7e6e7700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:59.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d7e6e7700 4 set_mon_vals no callback set 2023-09-30T00:47:59.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d76ffd700 1 -- 172.21.15.40:0/912674331 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6800bce0 con 0x7f1d780b1460 2023-09-30T00:47:59.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d76ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d76ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:59.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.445+0000 7f1d76ffd700 10 monclient: dump: 2023-09-30T00:47:59.449 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.449 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.449 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.449 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.449 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.449 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.449 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.449 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.449 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.449 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.446+0000 7f1d7e6e7700 10 monclient: shutdown 2023-09-30T00:47:59.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.446+0000 7f1d7e6e7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.446+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/912674331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d780b1460 msgr2=0x7f1d780b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.446+0000 7f1d7e6e7700 1 --2- 172.21.15.40:0/912674331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d780b1460 0x7f1d780b1830 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1d680099b0 tx=0x7f1d68009250 comp rx=0 tx=0).stop 2023-09-30T00:47:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.446+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/912674331 shutdown_connections 2023-09-30T00:47:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.446+0000 7f1d7e6e7700 1 --2- 172.21.15.40:0/912674331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d780b1460 0x7f1d780b1830 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.446+0000 7f1d7e6e7700 1 --2- 172.21.15.40:0/912674331 >> v2:172.21.15.135:3300/0 conn(0x7f1d780a5fb0 0x7f1d78141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.446+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/912674331 >> 172.21.15.40:0/912674331 conn(0x7f1d780631f0 msgr2=0x7f1d780635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.446+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/912674331 shutdown_connections 2023-09-30T00:47:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.446+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/912674331 wait complete. 2023-09-30T00:47:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.447+0000 7f1d7e6e7700 10 monclient: build_initial_monmap 2023-09-30T00:47:59.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.447+0000 7f1d7e6e7700 10 monclient: monmap: 2023-09-30T00:47:59.454 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:59.454 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:59.454 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:59.454 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:59.454 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:59.454 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:59.454 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:59.454 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:59.454 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:59.454 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.447+0000 7f1d7e6e7700 1 librados: starting msgr at 2023-09-30T00:47:59.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.447+0000 7f1d7e6e7700 1 librados: starting objecter 2023-09-30T00:47:59.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.447+0000 7f1d7e6e7700 1 Processor -- start 2023-09-30T00:47:59.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.447+0000 7f1d7e6e7700 1 -- start start 2023-09-30T00:47:59.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.447+0000 7f1d7e6e7700 1 librados: setting wanted keys 2023-09-30T00:47:59.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.447+0000 7f1d7e6e7700 1 librados: calling monclient init 2023-09-30T00:47:59.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.447+0000 7f1d7e6e7700 10 monclient: init 2023-09-30T00:47:59.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:59.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:47:59.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d780a5fb0 0x7f1d780a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 10 monclient(hunting): picked mon.noname-a con 0x7f1d780a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:59.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d780af410 0x7f1d7813ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 10 monclient(hunting): picked mon.noname-c con 0x7f1d780af410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:59.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1d780b1460 0x7f1d7814d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 10 monclient(hunting): picked mon.noname-b con 0x7f1d780b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:59.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d780a9e50 con 0x7f1d780a5fb0 2023-09-30T00:47:59.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d780aa2e0 con 0x7f1d780b1460 2023-09-30T00:47:59.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7813e990 con 0x7f1d780af410 2023-09-30T00:47:59.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:59.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.448+0000 7f1d7e6e7700 10 monclient(hunting): authenticate will time out at 5340.749424s 2023-09-30T00:47:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.449+0000 7f1d777fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d780af410 0x7f1d7813ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.450+0000 7f1d77fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d780a5fb0 0x7f1d780a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.450+0000 7f1d777fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d780af410 0x7f1d7813ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36706/0 (socket says 172.21.15.40:36706) 2023-09-30T00:47:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.450+0000 7f1d777fe700 1 -- 172.21.15.40:0/3436642936 learned_addr learned my addr 172.21.15.40:0/3436642936 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.450+0000 7f1d777fe700 10 monclient(hunting): get_auth_request con 0x7f1d780af410 auth_method 0 2023-09-30T00:47:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.450+0000 7f1d777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.450+0000 7f1d777fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.450+0000 7f1d777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.450+0000 7f1d77fff700 10 monclient(hunting): get_auth_request con 0x7f1d780a5fb0 auth_method 0 2023-09-30T00:47:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.450+0000 7f1d77fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.450+0000 7f1d77fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.450+0000 7f1d77fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:59.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.451+0000 7f1d777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:59.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.451+0000 7f1d777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:59.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.451+0000 7f1d777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:59.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.451+0000 7f1d77fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:59.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.451+0000 7f1d77fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:59.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.451+0000 7f1d77fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:59.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.456+0000 7f1d777fe700 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2023-09-30T00:47:59.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.456+0000 7f1d777fe700 1 -- 172.21.15.40:0/3436642936 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1d780b1460 msgr2=0x7f1d7814d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.456+0000 7f1d777fe700 1 --2- 172.21.15.40:0/3436642936 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1d780b1460 0x7f1d7814d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.456+0000 7f1d777fe700 1 -- 172.21.15.40:0/3436642936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d780a5fb0 msgr2=0x7f1d780a8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.456+0000 7f1d777fe700 1 --2- 172.21.15.40:0/3436642936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d780a5fb0 0x7f1d780a8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.456+0000 7f1d777fe700 10 monclient: _finish_hunting 0 2023-09-30T00:47:59.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.456+0000 7f1d777fe700 1 monclient: found mon.noname-c 2023-09-30T00:47:59.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.456+0000 7f1d777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:59.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.456+0000 7f1d777fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:47:59.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.456+0000 7f1d777fe700 1 -- 172.21.15.40:0/3436642936 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d680094f0 con 0x7f1d780af410 2023-09-30T00:47:59.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.459+0000 7f1d777fe700 1 --2- 172.21.15.40:0/3436642936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d780af410 0x7f1d7813ddb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00b810 tx=0x7f1d6c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:59.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.459+0000 7f1d5ffff700 10 client.?.objecter ms_handle_connect 0x7f1d780af410 2023-09-30T00:47:59.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.459+0000 7f1d5ffff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:59.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d5ffff700 1 -- 172.21.15.40:0/3436642936 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c00f510 con 0x7f1d780af410 2023-09-30T00:47:59.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d5ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d5ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:59.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d5ffff700 10 monclient: dump: 2023-09-30T00:47:59.466 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.466 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.466 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.466 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.466 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.466 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.466 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.467 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d5ffff700 10 monclient: _finish_auth 0 2023-09-30T00:47:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d5ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d5ffff700 1 -- 172.21.15.40:0/3436642936 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6c00f690 con 0x7f1d780af410 2023-09-30T00:47:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d5ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d5ffff700 1 -- 172.21.15.40:0/3436642936 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c00fa50 con 0x7f1d780af410 2023-09-30T00:47:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d7e6e7700 5 monclient: authenticate success, global_id 4301 2023-09-30T00:47:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d7e6e7700 10 monclient: _renew_subs 2023-09-30T00:47:59.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d7e6e7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:59.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/3436642936 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d780a9970 con 0x7f1d780af410 2023-09-30T00:47:59.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d767fc700 4 set_mon_vals no callback set 2023-09-30T00:47:59.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d5ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d5ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:59.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d5ffff700 10 monclient: dump: 2023-09-30T00:47:59.468 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.468 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.468 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.468 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.468 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.468 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.468 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.469 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.469 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.469 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d7e6e7700 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:59.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d7e6e7700 10 monclient: _renew_subs 2023-09-30T00:47:59.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d7e6e7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:59.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/3436642936 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7814de90 con 0x7f1d780af410 2023-09-30T00:47:59.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.464+0000 7f1d7e6e7700 1 librados: init done 2023-09-30T00:47:59.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.465+0000 7f1d7e6e7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:59.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.465+0000 7f1d7e6e7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:59.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.465+0000 7f1d7e6e7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:59.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.465+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/3436642936 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d780a9fc0 con 0x7f1d780af410 2023-09-30T00:47:59.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.496+0000 7fe97f8cd700 10 monclient: get_monmap_and_config 2023-09-30T00:47:59.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.496+0000 7fe97f8cd700 10 monclient: build_initial_monmap 2023-09-30T00:47:59.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.496+0000 7fe97f8cd700 10 monclient: monmap: 2023-09-30T00:47:59.497 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:59.497 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.497 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:59.497495+0000 2023-09-30T00:47:59.498 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:59.497495+0000 2023-09-30T00:47:59.498 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:59.498 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.498 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:59.498 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:59.498 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:59.498 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:59.498 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:59.498 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.497+0000 7fe97f8cd700 1 Processor -- start 2023-09-30T00:47:59.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.497+0000 7fe97f8cd700 1 -- start start 2023-09-30T00:47:59.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.497+0000 7fe97f8cd700 10 monclient: init 2023-09-30T00:47:59.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.497+0000 7fe97f8cd700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:59.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.497+0000 7fe97f8cd700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-09-30T00:47:59.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.497+0000 7fe97f8cd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9780b1460 0x7fe9780b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.497+0000 7fe97f8cd700 10 monclient(hunting): picked mon.noname-b con 0x7fe9780b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:59.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.497+0000 7fe97f8cd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9780a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:59.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.498+0000 7fe97f8cd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe97813ac20 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:59.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.498+0000 7fe97f8cd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9780a5b20 con 0x7fe97813ac20 2023-09-30T00:47:59.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.498+0000 7fe97f8cd700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9780a71f0 con 0x7fe9780a5fb0 2023-09-30T00:47:59.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.498+0000 7fe97f8cd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.498+0000 7fe97f8cd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780a74d0 con 0x7fe9780b1460 2023-09-30T00:47:59.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.498+0000 7fe97f8cd700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:59.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.498+0000 7fe97f8cd700 10 monclient(hunting): authenticate will time out at 5340.799072s 2023-09-30T00:47:59.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.498+0000 7fe97de6a700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe97813ac20 0x7fe97813b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43406/0 (socket says 172.21.15.40:43406) 2023-09-30T00:47:59.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.498+0000 7fe97de6a700 1 -- 172.21.15.40:0/3671973295 learned_addr learned my addr 172.21.15.40:0/3671973295 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:59.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.502+0000 7fe977fff700 1 -- 172.21.15.40:0/3671973295 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943876831 0 0) 0x7fe9780a5b20 con 0x7fe97813ac20 2023-09-30T00:47:59.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.502+0000 7fe977fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.502+0000 7fe977fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:59.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.502+0000 7fe977fff700 10 monclient(hunting): my global_id is 4329 2023-09-30T00:47:59.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.502+0000 7fe977fff700 1 -- 172.21.15.40:0/3671973295 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe960003570 con 0x7fe97813ac20 2023-09-30T00:47:59.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 -- 172.21.15.40:0/3671973295 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4198617939 0 0) 0x7fe960003570 con 0x7fe97813ac20 2023-09-30T00:47:59.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 -- 172.21.15.40:0/3671973295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9780b1460 msgr2=0x7fe9780b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:59.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 --2- 172.21.15.40:0/3671973295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9780b1460 0x7fe9780b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 -- 172.21.15.40:0/3671973295 >> v1:172.21.15.135:6789/0 conn(0x7fe9780a5fb0 legacy=0x7fe978141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:59.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 10 monclient: _finish_hunting 0 2023-09-30T00:47:59.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 monclient: found mon.noname-a-legacy 2023-09-30T00:47:59.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:59.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2023-09-30T00:47:59.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 -- 172.21.15.40:0/3671973295 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9780a63c0 con 0x7fe97813ac20 2023-09-30T00:47:59.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 10 monclient: _finish_auth 0 2023-09-30T00:47:59.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:59.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 -- 172.21.15.40:0/3671973295 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3033000540 0 0) 0x7fe9700017f0 con 0x7fe97813ac20 2023-09-30T00:47:59.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T00:47:59.509 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 10 monclient: dump: 2023-09-30T00:47:59.509 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.509 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.509 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.509 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.509 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.509 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.509 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.509 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.509 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.509 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.509 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2023-09-30T00:47:59.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:59.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 -- 172.21.15.40:0/3671973295 >> v1:172.21.15.40:6789/0 conn(0x7fe97813ac20 legacy=0x7fe97813b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:47:59.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 --2- 172.21.15.40:0/3671973295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe960003b10 0x7fe960003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 10 monclient(hunting): picked mon.b con 0x7fe960003b10 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:59.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 --2- 172.21.15.40:0/3671973295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe960004810 0x7fe960004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 10 monclient(hunting): picked mon.a con 0x7fe960004810 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:59.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 --2- 172.21.15.40:0/3671973295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9780b1460 0x7fe96000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 10 monclient(hunting): picked mon.c con 0x7fe9780b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:59.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 -- 172.21.15.40:0/3671973295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780a74d0 con 0x7fe960004810 2023-09-30T00:47:59.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 -- 172.21.15.40:0/3671973295 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe960005620 con 0x7fe960003b10 2023-09-30T00:47:59.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.506+0000 7fe977fff700 1 -- 172.21.15.40:0/3671973295 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe96000bfa0 con 0x7fe9780b1460 2023-09-30T00:47:59.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.507+0000 7fe977fff700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:59.511 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.507+0000 7fe977fff700 10 monclient(hunting): _finish_auth 0 2023-09-30T00:47:59.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.511+0000 7fe97ce68700 1 --2- 172.21.15.40:0/3671973295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe960004810 0x7fe960004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.512+0000 7fe97ce68700 10 monclient(hunting): get_auth_request con 0x7fe960004810 auth_method 0 2023-09-30T00:47:59.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.512+0000 7fe97ce68700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.512+0000 7fe97ce68700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.512+0000 7fe97ce68700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:47:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.512+0000 7fe97ce68700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.512+0000 7fe97ce68700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:59.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.512+0000 7fe97ce68700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:47:59.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.515+0000 7f1d5ffff700 1 -- 172.21.15.40:0/3436642936 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1d6c024430 con 0x7f1d780af410 2023-09-30T00:47:59.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.515+0000 7f1d5ffff700 1 --2- 172.21.15.40:0/3436642936 >> v2:172.21.15.135:6800/116426 conn(0x7f1d64068960 0x7f1d6406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.515+0000 7f1d5ffff700 1 -- 172.21.15.40:0/3436642936 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1d6c084470 con 0x7f1d780af410 2023-09-30T00:47:59.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.515+0000 7f1d5ffff700 10 client.4301.objecter ms_dispatch 0x7f1d78001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:59.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.515+0000 7f1d5ffff700 3 client.4301.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:59.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.515+0000 7f1d5ffff700 3 client.4301.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:59.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.515+0000 7f1d5ffff700 20 client.4301.objecter dump_active .. 0 homeless 2023-09-30T00:47:59.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.515+0000 7f1d77fff700 1 --2- 172.21.15.40:0/3436642936 >> v2:172.21.15.135:6800/116426 conn(0x7f1d64068960 0x7f1d6406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.515+0000 7f1d77fff700 10 monclient: get_auth_request con 0x7f1d64068960 auth_method 0 2023-09-30T00:47:59.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7f1d5ffff700 10 client.4301.objecter ms_handle_connect 0x7f1d64068960 2023-09-30T00:47:59.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7f1d77fff700 1 --2- 172.21.15.40:0/3436642936 >> v2:172.21.15.135:6800/116426 conn(0x7f1d64068960 0x7f1d6406ae10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d680099b0 tx=0x7f1d68016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:59.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7fe97ce68700 10 monclient(hunting): handle_auth_done global_id 4329 payload 293 2023-09-30T00:47:59.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7fe97ce68700 1 -- 172.21.15.40:0/3671973295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9780b1460 msgr2=0x7fe96000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:59.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7fe97ce68700 1 --2- 172.21.15.40:0/3671973295 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9780b1460 0x7fe96000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7fe97ce68700 1 -- 172.21.15.40:0/3671973295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe960003b10 msgr2=0x7fe960003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:59.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7fe97ce68700 1 --2- 172.21.15.40:0/3671973295 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe960003b10 0x7fe960003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.518 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7fe97ce68700 10 monclient: _finish_hunting 0 2023-09-30T00:47:59.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7fe97ce68700 1 monclient: found mon.a 2023-09-30T00:47:59.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7fe97ce68700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:59.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7fe97ce68700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:59.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7fe97ce68700 1 -- 172.21.15.40:0/3671973295 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe960003570 con 0x7fe960004810 2023-09-30T00:47:59.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7fe97ce68700 10 monclient: _finish_auth 0 2023-09-30T00:47:59.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7fe97ce68700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:59.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7fe97f8cd700 5 monclient: authenticate success, global_id 4329 2023-09-30T00:47:59.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.516+0000 7fe97f8cd700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:59.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.518+0000 7fe97ce68700 1 --2- 172.21.15.40:0/3671973295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe960004810 0x7fe960004c80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe96c0017d0 tx=0x7fe96c007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c875f14c4c950214 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:59.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe977fff700 1 -- 172.21.15.40:0/3671973295 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96c012400 con 0x7fe960004810 2023-09-30T00:47:59.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe977fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe977fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:59.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe977fff700 10 monclient: dump: 2023-09-30T00:47:59.524 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.524 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.524 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.524 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.524 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.524 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.524 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.524 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.524 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.524 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe977fff700 1 -- 172.21.15.40:0/3671973295 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96c012580 con 0x7fe960004810 2023-09-30T00:47:59.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe97f8cd700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:59.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe977fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:59.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe97f8cd700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:59.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe977fff700 1 -- 172.21.15.40:0/3671973295 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96c012940 con 0x7fe960004810 2023-09-30T00:47:59.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe977fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe97f8cd700 4 set_mon_vals no callback set 2023-09-30T00:47:59.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe977fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:59.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe977fff700 10 monclient: dump: 2023-09-30T00:47:59.525 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.525 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.526 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.526 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.526 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.526 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.526 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.526 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.526 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.526 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe97f8cd700 10 monclient: shutdown 2023-09-30T00:47:59.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe97f8cd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:59.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe97f8cd700 1 -- 172.21.15.40:0/3671973295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe960004810 msgr2=0x7fe960004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.528 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe97f8cd700 1 --2- 172.21.15.40:0/3671973295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe960004810 0x7fe960004c80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe96c0017d0 tx=0x7fe96c007d10 comp rx=0 tx=0).stop 2023-09-30T00:47:59.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe97de6a700 1 -- 172.21.15.40:0/3671973295 reap_dead start 2023-09-30T00:47:59.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe97f8cd700 1 -- 172.21.15.40:0/3671973295 shutdown_connections 2023-09-30T00:47:59.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.523+0000 7fe97f8cd700 1 -- 172.21.15.40:0/3671973295 >> 172.21.15.40:0/3671973295 conn(0x7fe9780631f0 msgr2=0x7fe9780635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:59.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.524+0000 7fe97f8cd700 1 -- 172.21.15.40:0/3671973295 shutdown_connections 2023-09-30T00:47:59.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.524+0000 7fe97f8cd700 1 -- 172.21.15.40:0/3671973295 wait complete. 2023-09-30T00:47:59.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.524+0000 7fe97f8cd700 10 monclient: build_initial_monmap 2023-09-30T00:47:59.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.524+0000 7fe97f8cd700 10 monclient: monmap: 2023-09-30T00:47:59.529 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:59.529 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:59.529 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:59.529 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:59.529 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:59.530 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.524+0000 7fe97f8cd700 1 librados: starting msgr at 2023-09-30T00:47:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.524+0000 7fe97f8cd700 1 librados: starting objecter 2023-09-30T00:47:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 1 Processor -- start 2023-09-30T00:47:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 1 -- start start 2023-09-30T00:47:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 1 librados: setting wanted keys 2023-09-30T00:47:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 1 librados: calling monclient init 2023-09-30T00:47:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 10 monclient: init 2023-09-30T00:47:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:47:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe960004810 0x7fe97813e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 10 monclient(hunting): picked mon.noname-b con 0x7fe960004810 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9780a9910 0x7fe9780a9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 10 monclient(hunting): picked mon.noname-c con 0x7fe9780a9910 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9780ae270 0x7fe9780ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 10 monclient(hunting): picked mon.noname-a con 0x7fe9780ae270 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780aa7c0 con 0x7fe9780ae270 2023-09-30T00:47:59.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780aa4a0 con 0x7fe960004810 2023-09-30T00:47:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780aec20 con 0x7fe9780a9910 2023-09-30T00:47:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.525+0000 7fe97f8cd700 10 monclient(hunting): authenticate will time out at 5340.826973s 2023-09-30T00:47:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.526+0000 7fe97de6a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9780ae270 0x7fe9780ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.526+0000 7fe97de6a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9780ae270 0x7fe9780ae6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56732/0 (socket says 172.21.15.40:56732) 2023-09-30T00:47:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.526+0000 7fe97de6a700 1 -- 172.21.15.40:0/1369260279 learned_addr learned my addr 172.21.15.40:0/1369260279 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.526+0000 7fe97de6a700 10 monclient(hunting): get_auth_request con 0x7fe9780ae270 auth_method 0 2023-09-30T00:47:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.526+0000 7fe97de6a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.526+0000 7fe97de6a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.526+0000 7fe97de6a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.527+0000 7fe97de6a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.527+0000 7fe97de6a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.527+0000 7fe97de6a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.531+0000 7fe97de6a700 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2023-09-30T00:47:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.531+0000 7fe97de6a700 1 -- 172.21.15.40:0/1369260279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9780a9910 msgr2=0x7fe9780a9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.531+0000 7fe97de6a700 1 --2- 172.21.15.40:0/1369260279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9780a9910 0x7fe9780a9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.531+0000 7fe97de6a700 1 -- 172.21.15.40:0/1369260279 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe960004810 msgr2=0x7fe97813e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.531+0000 7fe97de6a700 1 --2- 172.21.15.40:0/1369260279 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe960004810 0x7fe97813e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.531+0000 7fe97de6a700 10 monclient: _finish_hunting 0 2023-09-30T00:47:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.531+0000 7fe97de6a700 1 monclient: found mon.noname-a 2023-09-30T00:47:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.531+0000 7fe97de6a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.531+0000 7fe97de6a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:47:59.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.531+0000 7fe97de6a700 1 -- 172.21.15.40:0/1369260279 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9780aede0 con 0x7fe9780ae270 2023-09-30T00:47:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.534+0000 7fe97de6a700 1 --2- 172.21.15.40:0/1369260279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9780ae270 0x7fe9780ae6e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe9780b0d20 tx=0x7fe970007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.534+0000 7fe9757fa700 10 client.?.objecter ms_handle_connect 0x7fe9780ae270 2023-09-30T00:47:59.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.534+0000 7fe9757fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:59.540 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.539+0000 7fe9757fa700 1 -- 172.21.15.40:0/1369260279 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe970004dc0 con 0x7fe9780ae270 2023-09-30T00:47:59.540 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.539+0000 7fe9757fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.541 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.539+0000 7fe9757fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:59.541 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.539+0000 7fe9757fa700 10 monclient: dump: 2023-09-30T00:47:59.541 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.541 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.541 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.541 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.541 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.541 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.541 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.541 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.541 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.542 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.539+0000 7fe9757fa700 10 monclient: _finish_auth 0 2023-09-30T00:47:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.539+0000 7fe9757fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.539+0000 7fe9757fa700 1 -- 172.21.15.40:0/1369260279 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97000a700 con 0x7fe9780ae270 2023-09-30T00:47:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.539+0000 7fe9757fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.539+0000 7fe9757fa700 1 -- 172.21.15.40:0/1369260279 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97000aac0 con 0x7fe9780ae270 2023-09-30T00:47:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.539+0000 7fe97f8cd700 5 monclient: authenticate success, global_id 4332 2023-09-30T00:47:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.539+0000 7fe97f8cd700 10 monclient: _renew_subs 2023-09-30T00:47:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.539+0000 7fe97f8cd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.539+0000 7fe97f8cd700 1 -- 172.21.15.40:0/1369260279 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe978150cc0 con 0x7fe9780ae270 2023-09-30T00:47:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.540+0000 7fe9757fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.542 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.540+0000 7fe9757fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.539+0000 7fe9777fe700 4 set_mon_vals no callback set 2023-09-30T00:47:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.540+0000 7fe9757fa700 10 monclient: dump: 2023-09-30T00:47:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.543 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.543 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.540+0000 7fe97f8cd700 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.540+0000 7fe97f8cd700 10 monclient: _renew_subs 2023-09-30T00:47:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.540+0000 7fe97f8cd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.540+0000 7fe97f8cd700 1 -- 172.21.15.40:0/1369260279 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe978151130 con 0x7fe9780ae270 2023-09-30T00:47:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.540+0000 7fe97f8cd700 1 librados: init done 2023-09-30T00:47:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.541+0000 7fe97f8cd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.541+0000 7fe97f8cd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:59.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.541+0000 7fe97f8cd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:47:59.545 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.541+0000 7fe97f8cd700 1 -- 172.21.15.40:0/1369260279 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9780aa900 con 0x7fe9780ae270 2023-09-30T00:47:59.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.590+0000 7f52fd7fa700 1 -- 172.21.15.40:0/788429292 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5300011390 con 0x7f53080ae830 2023-09-30T00:47:59.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.590+0000 7f52fd7fa700 10 client.4357.objecter ms_dispatch 0x7f5308001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:59.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.590+0000 7f52fd7fa700 3 client.4357.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:59.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.590+0000 7f52fd7fa700 3 client.4357.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:59.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.590+0000 7f52fd7fa700 20 client.4357.objecter dump_active .. 0 homeless 2023-09-30T00:47:59.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.614+0000 7f52fd7fa700 1 -- 172.21.15.40:0/788429292 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5300053670 con 0x7f53080ae830 2023-09-30T00:47:59.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.614+0000 7f52fd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:59.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.614+0000 7f52fd7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:59.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:47:59.628 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:47:59.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.681+0000 7fe9757fa700 1 -- 172.21.15.40:0/1369260279 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fe970007190 con 0x7fe9780ae270 2023-09-30T00:47:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.682+0000 7fe9757fa700 1 --2- 172.21.15.40:0/1369260279 >> v2:172.21.15.135:6800/116426 conn(0x7fe964068890 0x7fe96406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.682+0000 7fe9757fa700 1 -- 172.21.15.40:0/1369260279 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fe970083f10 con 0x7fe9780ae270 2023-09-30T00:47:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.682+0000 7fe9757fa700 10 client.4332.objecter ms_dispatch 0x7fe978001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:47:59.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.682+0000 7fe9757fa700 3 client.4332.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:47:59.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.682+0000 7fe9757fa700 3 client.4332.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:47:59.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.682+0000 7fe9757fa700 20 client.4332.objecter dump_active .. 0 homeless 2023-09-30T00:47:59.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.682+0000 7fe97d669700 1 --2- 172.21.15.40:0/1369260279 >> v2:172.21.15.135:6800/116426 conn(0x7fe964068890 0x7fe96406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.682+0000 7fe97d669700 10 monclient: get_auth_request con 0x7fe964068890 auth_method 0 2023-09-30T00:47:59.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.683+0000 7fe9757fa700 10 client.4332.objecter ms_handle_connect 0x7fe964068890 2023-09-30T00:47:59.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.683+0000 7fe97d669700 1 --2- 172.21.15.40:0/1369260279 >> v2:172.21.15.135:6800/116426 conn(0x7fe964068890 0x7fe96406ad40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe968006fd0 tx=0x7fe968006d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:59.743 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T00:47:59.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.799+0000 7f22e85fe700 10 monclient: get_monmap_and_config 2023-09-30T00:47:59.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.799+0000 7f22e85fe700 10 monclient: build_initial_monmap 2023-09-30T00:47:59.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.799+0000 7f22e85fe700 10 monclient: monmap: 2023-09-30T00:47:59.800 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:59.800 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.800 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:59.800475+0000 2023-09-30T00:47:59.800 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:59.800475+0000 2023-09-30T00:47:59.800 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:59.800 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.801 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:59.801 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:59.801 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:59.801 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:59.801 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:59.801 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.801+0000 7f22e85fe700 1 Processor -- start 2023-09-30T00:47:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.801+0000 7f22e85fe700 1 -- start start 2023-09-30T00:47:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.801+0000 7f22e85fe700 10 monclient: init 2023-09-30T00:47:59.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.802+0000 7f22e85fe700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:59.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.802+0000 7f22e85fe700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-09-30T00:47:59.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.802+0000 7f22e85fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f22e00b14a0 0x7f22e00afe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.802+0000 7f22e85fe700 10 monclient(hunting): picked mon.noname-a con 0x7f22e00b14a0 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:59.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.802+0000 7f22e85fe700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22e00b1870 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:59.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.802+0000 7f22e85fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22e006dd20 0x7f22e006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.802+0000 7f22e85fe700 10 monclient(hunting): picked mon.noname-b con 0x7f22e006dd20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:59.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.802+0000 7f22e85fe700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22e013ff40 con 0x7f22e00b1870 2023-09-30T00:47:59.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.802+0000 7f22e85fe700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.802+0000 7f22e85fe700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22e006b030 con 0x7f22e00b14a0 2023-09-30T00:47:59.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.802+0000 7f22e85fe700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.802+0000 7f22e85fe700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e006b310 con 0x7f22e006dd20 2023-09-30T00:47:59.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.802+0000 7f22e85fe700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:59.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.802+0000 7f22e85fe700 10 monclient(hunting): authenticate will time out at 5341.103417s 2023-09-30T00:47:59.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.803+0000 7f22e5b99700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f22e00b1870 0x7f22e0145b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:54076/0 (socket says 172.21.15.40:54076) 2023-09-30T00:47:59.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.803+0000 7f22e5b99700 1 -- 172.21.15.40:0/2233428570 learned_addr learned my addr 172.21.15.40:0/2233428570 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:47:59.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.806+0000 7f22e5398700 1 -- 172.21.15.40:0/2233428570 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801773823 0 0) 0x7f22e013ff40 con 0x7f22e00b1870 2023-09-30T00:47:59.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.806+0000 7f22e5398700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.806+0000 7f22e5398700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:59.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.806+0000 7f22e5398700 10 monclient(hunting): my global_id is 4360 2023-09-30T00:47:59.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.806+0000 7f22e5398700 1 -- 172.21.15.40:0/2233428570 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22cc003570 con 0x7f22e00b1870 2023-09-30T00:47:59.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 1 -- 172.21.15.40:0/2233428570 <== mon.1 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4222376482 0 0) 0x7f22cc003570 con 0x7f22e00b1870 2023-09-30T00:47:59.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 1 -- 172.21.15.40:0/2233428570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22e006dd20 msgr2=0x7f22e006e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:59.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 1 --2- 172.21.15.40:0/2233428570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22e006dd20 0x7f22e006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 1 -- 172.21.15.40:0/2233428570 >> v2:172.21.15.40:3300/0 conn(0x7f22e00b14a0 msgr2=0x7f22e00afe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:59.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 1 --2- 172.21.15.40:0/2233428570 >> v2:172.21.15.40:3300/0 conn(0x7f22e00b14a0 0x7f22e00afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 10 monclient: _finish_hunting 0 2023-09-30T00:47:59.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 1 monclient: found mon.noname-c-legacy 2023-09-30T00:47:59.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:59.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.135:6789/0 2023-09-30T00:47:59.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 1 -- 172.21.15.40:0/2233428570 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e00af8f0 con 0x7f22e00b1870 2023-09-30T00:47:59.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 10 monclient: _finish_auth 0 2023-09-30T00:47:59.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:59.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 1 -- 172.21.15.40:0/2233428570 <== mon.1 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3033000540 0 0) 0x7f22dc001470 con 0x7f22e00b1870 2023-09-30T00:47:59.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-09-30T00:47:59.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 10 monclient: dump: 2023-09-30T00:47:59.812 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.813 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] but i'm connected to v1:172.21.15.135:6789/0, reconnecting 2023-09-30T00:47:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 1 -- 172.21.15.40:0/2233428570 >> v1:172.21.15.135:6789/0 conn(0x7f22e00b1870 legacy=0x7f22e0145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:47:59.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 1 --2- 172.21.15.40:0/2233428570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22cc003ae0 0x7f22cc003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 10 monclient(hunting): picked mon.b con 0x7f22cc003ae0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:59.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 1 --2- 172.21.15.40:0/2233428570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22cc004810 0x7f22cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.810+0000 7f22e5398700 10 monclient(hunting): picked mon.c con 0x7f22cc004810 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:59.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.811+0000 7f22e5398700 1 --2- 172.21.15.40:0/2233428570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22cc005310 0x7f22cc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.811+0000 7f22e5398700 10 monclient(hunting): picked mon.a con 0x7f22cc005310 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:59.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.811+0000 7f22e5398700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.811+0000 7f22e5398700 1 -- 172.21.15.40:0/2233428570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e006b030 con 0x7f22cc005310 2023-09-30T00:47:59.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.811+0000 7f22e5398700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.811+0000 7f22e5398700 1 -- 172.21.15.40:0/2233428570 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e006b310 con 0x7f22cc003ae0 2023-09-30T00:47:59.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.811+0000 7f22e5398700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.811+0000 7f22e5398700 1 -- 172.21.15.40:0/2233428570 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22cc00c320 con 0x7f22cc004810 2023-09-30T00:47:59.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.811+0000 7f22e5398700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:59.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.811+0000 7f22e5398700 10 monclient(hunting): _finish_auth 0 2023-09-30T00:47:59.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.811+0000 7f22e639a700 1 --2- 172.21.15.40:0/2233428570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22cc003ae0 0x7f22cc003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.813+0000 7f22e639a700 10 monclient(hunting): get_auth_request con 0x7f22cc003ae0 auth_method 0 2023-09-30T00:47:59.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.813+0000 7f22e639a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:59.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.813+0000 7f22e639a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.813+0000 7f22e639a700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:47:59.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.813+0000 7f22e639a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:59.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.813+0000 7f22e639a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:59.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.813+0000 7f22e639a700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:47:59.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.814+0000 7ff8df01c700 10 monclient: get_monmap_and_config 2023-09-30T00:47:59.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.814+0000 7ff8df01c700 10 monclient: build_initial_monmap 2023-09-30T00:47:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.815+0000 7ff8df01c700 10 monclient: monmap: 2023-09-30T00:47:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:59.816041+0000 2023-09-30T00:47:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:59.816041+0000 2023-09-30T00:47:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:59.817 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.815+0000 7f22e639a700 10 monclient(hunting): handle_auth_done global_id 4360 payload 293 2023-09-30T00:47:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.816+0000 7f22e639a700 1 -- 172.21.15.40:0/2233428570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22cc004810 msgr2=0x7f22cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.816+0000 7f22e639a700 1 --2- 172.21.15.40:0/2233428570 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22cc004810 0x7f22cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.816+0000 7f22e639a700 1 -- 172.21.15.40:0/2233428570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22cc005310 msgr2=0x7f22cc00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.816+0000 7f22e639a700 1 --2- 172.21.15.40:0/2233428570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22cc005310 0x7f22cc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.816+0000 7f22e639a700 10 monclient: _finish_hunting 0 2023-09-30T00:47:59.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.816+0000 7f22e639a700 1 monclient: found mon.b 2023-09-30T00:47:59.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.816+0000 7f22e639a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:59.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.816+0000 7f22e639a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:59.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.816+0000 7f22e639a700 1 -- 172.21.15.40:0/2233428570 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e00af8f0 con 0x7f22cc003ae0 2023-09-30T00:47:59.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.816+0000 7f22e639a700 10 monclient: _finish_auth 0 2023-09-30T00:47:59.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.816+0000 7f22e639a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:59.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.816+0000 7f22e85fe700 5 monclient: authenticate success, global_id 4360 2023-09-30T00:47:59.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.816+0000 7f22e85fe700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:59.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.816+0000 7ff8df01c700 1 Processor -- start 2023-09-30T00:47:59.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 1 -- start start 2023-09-30T00:47:59.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 10 monclient: init 2023-09-30T00:47:59.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-09-30T00:47:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8d80af5f0 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8d8068b10 0x7ff8d8146070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 10 monclient(hunting): picked mon.noname-b con 0x7ff8d8068b10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff8d80adfe0 0x7ff8d81465b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 10 monclient(hunting): picked mon.noname-a con 0x7ff8d80adfe0 addr v2:172.21.15.40:3300/0 2023-09-30T00:47:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d80af9c0 con 0x7ff8d80af5f0 2023-09-30T00:47:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d806f170 con 0x7ff8d80adfe0 2023-09-30T00:47:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d80aeb00 con 0x7ff8d8068b10 2023-09-30T00:47:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:59.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.817+0000 7ff8df01c700 10 monclient(hunting): authenticate will time out at 5341.118894s 2023-09-30T00:47:59.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.818+0000 7f22e639a700 1 --2- 172.21.15.40:0/2233428570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22cc003ae0 0x7f22cc003ef0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f22d8009df0 tx=0x7f22d80095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5c334fa4e8bf57c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:59.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.822+0000 7f22e5398700 1 -- 172.21.15.40:0/2233428570 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d800bb90 con 0x7f22cc003ae0 2023-09-30T00:47:59.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.822+0000 7f22e5398700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.822+0000 7f22e5398700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:59.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.822+0000 7f22e5398700 10 monclient: dump: 2023-09-30T00:47:59.823 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.823 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.823 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.823 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.823 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.823 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.824 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.824 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.824 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.824 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.822+0000 7f22e5398700 1 -- 172.21.15.40:0/2233428570 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d800bcf0 con 0x7f22cc003ae0 2023-09-30T00:47:59.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.822+0000 7f22e5398700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:59.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.822+0000 7f22e5398700 1 -- 172.21.15.40:0/2233428570 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d8014690 con 0x7f22cc003ae0 2023-09-30T00:47:59.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.822+0000 7f22e5398700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.822+0000 7f22e5398700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:59.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.822+0000 7f22e5398700 10 monclient: dump: 2023-09-30T00:47:59.824 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.825 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.825 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.825 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.825 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.825 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.825 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.825 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.825 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.825 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.822+0000 7f22e85fe700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:59.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.822+0000 7f22e85fe700 4 set_mon_vals no callback set 2023-09-30T00:47:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.823+0000 7f22e85fe700 10 monclient: shutdown 2023-09-30T00:47:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.823+0000 7f22e85fe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.823+0000 7f22e85fe700 1 -- 172.21.15.40:0/2233428570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22cc003ae0 msgr2=0x7f22cc003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.823+0000 7f22e85fe700 1 --2- 172.21.15.40:0/2233428570 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22cc003ae0 0x7f22cc003ef0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f22d8009df0 tx=0x7f22d80095f0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.823+0000 7f22e6b9b700 1 -- 172.21.15.40:0/2233428570 reap_dead start 2023-09-30T00:47:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.823+0000 7f22e85fe700 1 -- 172.21.15.40:0/2233428570 shutdown_connections 2023-09-30T00:47:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.823+0000 7f22e85fe700 1 -- 172.21.15.40:0/2233428570 >> 172.21.15.40:0/2233428570 conn(0x7f22e00631f0 msgr2=0x7f22e00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.823+0000 7f22e85fe700 1 -- 172.21.15.40:0/2233428570 shutdown_connections 2023-09-30T00:47:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.823+0000 7f22e85fe700 1 -- 172.21.15.40:0/2233428570 wait complete. 2023-09-30T00:47:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.824+0000 7f22e85fe700 10 monclient: build_initial_monmap 2023-09-30T00:47:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.824+0000 7f22e85fe700 10 monclient: monmap: 2023-09-30T00:47:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:59.828 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.824+0000 7f22e85fe700 1 librados: starting msgr at 2023-09-30T00:47:59.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.824+0000 7f22e85fe700 1 librados: starting objecter 2023-09-30T00:47:59.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.824+0000 7f22e85fe700 1 Processor -- start 2023-09-30T00:47:59.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.824+0000 7f22e85fe700 1 -- start start 2023-09-30T00:47:59.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.824+0000 7f22e85fe700 1 librados: setting wanted keys 2023-09-30T00:47:59.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.824+0000 7f22e85fe700 1 librados: calling monclient init 2023-09-30T00:47:59.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.824+0000 7f22e85fe700 10 monclient: init 2023-09-30T00:47:59.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:59.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:47:59.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22cc003ae0 0x7f22e011c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 10 monclient(hunting): picked mon.noname-c con 0x7f22cc003ae0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:59.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22e011ce80 0x7f22e0155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 10 monclient(hunting): picked mon.noname-a con 0x7f22e011ce80 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:59.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22e0155d90 0x7f22e0152270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 10 monclient(hunting): picked mon.noname-b con 0x7f22e0155d90 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:59.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e011d2f0 con 0x7f22e011ce80 2023-09-30T00:47:59.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e0142fa0 con 0x7f22e0155d90 2023-09-30T00:47:59.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e0143280 con 0x7f22cc003ae0 2023-09-30T00:47:59.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:59.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.825+0000 7f22e85fe700 10 monclient(hunting): authenticate will time out at 5341.126343s 2023-09-30T00:47:59.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.826+0000 7f22e6b9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22e0155d90 0x7f22e0152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.826+0000 7f22e6b9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22e0155d90 0x7f22e0152270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43448/0 (socket says 172.21.15.40:43448) 2023-09-30T00:47:59.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.826+0000 7f22e6b9b700 1 -- 172.21.15.40:0/4129146734 learned_addr learned my addr 172.21.15.40:0/4129146734 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:59.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.826+0000 7f22e6b9b700 10 monclient(hunting): get_auth_request con 0x7f22e0155d90 auth_method 0 2023-09-30T00:47:59.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.826+0000 7f22e6b9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:59.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.826+0000 7f22e6b9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.826+0000 7f22e6b9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:59.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.826+0000 7f22e6b9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:59.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.826+0000 7f22e6b9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:59.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.827+0000 7f22e6b9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:59.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.830+0000 7f22e6b9b700 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2023-09-30T00:47:59.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.830+0000 7f22e6b9b700 1 -- 172.21.15.40:0/4129146734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22cc003ae0 msgr2=0x7f22e011c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.830+0000 7f22e6b9b700 1 --2- 172.21.15.40:0/4129146734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22cc003ae0 0x7f22e011c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.830+0000 7f22e6b9b700 1 -- 172.21.15.40:0/4129146734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22e011ce80 msgr2=0x7f22e0155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.830+0000 7f22e6b9b700 1 --2- 172.21.15.40:0/4129146734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22e011ce80 0x7f22e0155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.830+0000 7f22e6b9b700 10 monclient: _finish_hunting 0 2023-09-30T00:47:59.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.830+0000 7f22e6b9b700 1 monclient: found mon.noname-b 2023-09-30T00:47:59.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.830+0000 7f22e6b9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:59.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.830+0000 7f22e6b9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:47:59.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.830+0000 7f22e6b9b700 1 -- 172.21.15.40:0/4129146734 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e011d5c0 con 0x7f22e0155d90 2023-09-30T00:47:59.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.832+0000 7f22e6b9b700 1 --2- 172.21.15.40:0/4129146734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22e0155d90 0x7f22e0152270 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f22e006e0f0 tx=0x7f22d0002d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:59.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.832+0000 7f22d67fc700 10 client.?.objecter ms_handle_connect 0x7f22e0155d90 2023-09-30T00:47:59.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.832+0000 7f22d67fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:59.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22d67fc700 1 -- 172.21.15.40:0/4129146734 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d000d400 con 0x7f22e0155d90 2023-09-30T00:47:59.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22d67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:59.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22d67fc700 10 monclient: dump: 2023-09-30T00:47:59.837 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.837 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.837 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.837 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.837 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.837 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.837 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.837 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.837 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.837 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22d67fc700 10 monclient: _finish_auth 0 2023-09-30T00:47:59.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22d67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:59.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22d67fc700 1 -- 172.21.15.40:0/4129146734 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d0011070 con 0x7f22e0155d90 2023-09-30T00:47:59.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22d67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:59.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22d67fc700 1 -- 172.21.15.40:0/4129146734 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d000d5c0 con 0x7f22e0155d90 2023-09-30T00:47:59.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22d67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:59.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22d67fc700 10 monclient: dump: 2023-09-30T00:47:59.838 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.838 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.838 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.838 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.839 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.839 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.839 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.839 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.839 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.839 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22e85fe700 5 monclient: authenticate success, global_id 4363 2023-09-30T00:47:59.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22e85fe700 10 monclient: _renew_subs 2023-09-30T00:47:59.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22e85fe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:59.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22e85fe700 1 -- 172.21.15.40:0/4129146734 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e0142360 con 0x7f22e0155d90 2023-09-30T00:47:59.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22e85fe700 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:59.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22e85fe700 10 monclient: _renew_subs 2023-09-30T00:47:59.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22e85fe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:59.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22e85fe700 1 -- 172.21.15.40:0/4129146734 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e0156480 con 0x7f22e0155d90 2023-09-30T00:47:59.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.836+0000 7f22e85fe700 1 librados: init done 2023-09-30T00:47:59.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.837+0000 7f22e4b97700 4 set_mon_vals no callback set 2023-09-30T00:47:59.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.837+0000 7f22e85fe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:59.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.838+0000 7f22e85fe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:59.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.838+0000 7f22e85fe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:47:59.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.838+0000 7f22e85fe700 1 -- 172.21.15.40:0/4129146734 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c80050b0 con 0x7f22e0155d90 2023-09-30T00:47:59.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.841+0000 7f1d5ffff700 1 -- 172.21.15.40:0/3436642936 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d6c051f40 con 0x7f1d780af410 2023-09-30T00:47:59.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.841+0000 7f1d5ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:59.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.841+0000 7f1d5ffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:59.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.844+0000 7ff8d7fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8d8068b10 0x7ff8d8146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.846+0000 7ff8d7fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8d8068b10 0x7ff8d8146070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36744/0 (socket says 172.21.15.40:36744) 2023-09-30T00:47:59.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.846+0000 7ff8d7fff700 1 -- 172.21.15.40:0/814499531 learned_addr learned my addr 172.21.15.40:0/814499531 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:59.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.846+0000 7ff8d7fff700 10 monclient(hunting): get_auth_request con 0x7ff8d8068b10 auth_method 0 2023-09-30T00:47:59.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.846+0000 7ff8d7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:59.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.846+0000 7ff8d7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.846+0000 7ff8d7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:59.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.847+0000 7ff8d7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:59.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.847+0000 7ff8d7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:59.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.847+0000 7ff8d7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:59.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.851+0000 7ff8d7fff700 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2023-09-30T00:47:59.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.851+0000 7ff8d7fff700 1 -- 172.21.15.40:0/814499531 >> v2:172.21.15.40:3300/0 conn(0x7ff8d80adfe0 msgr2=0x7ff8d81465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:59.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.851+0000 7ff8d7fff700 1 --2- 172.21.15.40:0/814499531 >> v2:172.21.15.40:3300/0 conn(0x7ff8d80adfe0 0x7ff8d81465b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.851+0000 7ff8d7fff700 1 -- 172.21.15.40:0/814499531 >> v1:172.21.15.40:6789/0 conn(0x7ff8d80af5f0 legacy=0x7ff8d80ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:59.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.851+0000 7ff8d7fff700 10 monclient: _finish_hunting 0 2023-09-30T00:47:59.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.851+0000 7ff8d7fff700 1 monclient: found mon.noname-b 2023-09-30T00:47:59.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.851+0000 7ff8d7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:59.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.851+0000 7ff8d7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:47:59.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.851+0000 7ff8d7fff700 1 -- 172.21.15.40:0/814499531 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d80aede0 con 0x7ff8d8068b10 2023-09-30T00:47:59.858 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T00:47:59.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.858+0000 7ff8d7fff700 1 --2- 172.21.15.40:0/814499531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8d8068b10 0x7ff8d8146070 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40099b0 tx=0x7ff8c4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77ff36d1c8d51209 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:59.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.864+0000 7ff8d77fe700 1 -- 172.21.15.40:0/814499531 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c400b820 con 0x7ff8d8068b10 2023-09-30T00:47:59.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.864+0000 7ff8d77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.864+0000 7ff8d77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:59.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.864+0000 7ff8d77fe700 10 monclient: dump: 2023-09-30T00:47:59.865 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.865 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.865 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.865 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.865 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.865 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.865 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.865 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.865 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.865 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.864+0000 7ff8d77fe700 10 monclient: _finish_auth 0 2023-09-30T00:47:59.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.864+0000 7ff8d77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:59.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.864+0000 7ff8d77fe700 1 -- 172.21.15.40:0/814499531 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c400b980 con 0x7ff8d8068b10 2023-09-30T00:47:59.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.864+0000 7ff8d77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:59.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.864+0000 7ff8d77fe700 1 -- 172.21.15.40:0/814499531 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c400bd20 con 0x7ff8d8068b10 2023-09-30T00:47:59.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.864+0000 7ff8d77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.864+0000 7ff8d77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:59.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.864+0000 7ff8d77fe700 10 monclient: dump: 2023-09-30T00:47:59.866 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.866 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.867 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.867 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.867 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.867 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.867 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.867 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.867 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.867 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.865+0000 7ff8df01c700 5 monclient: authenticate success, global_id 4304 2023-09-30T00:47:59.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.865+0000 7ff8df01c700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:59.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.865+0000 7ff8df01c700 4 set_mon_vals no callback set 2023-09-30T00:47:59.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.866+0000 7ff8df01c700 10 monclient: shutdown 2023-09-30T00:47:59.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.866+0000 7ff8df01c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:59.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.866+0000 7ff8df01c700 1 -- 172.21.15.40:0/814499531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8d8068b10 msgr2=0x7ff8d8146070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.866+0000 7ff8df01c700 1 --2- 172.21.15.40:0/814499531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8d8068b10 0x7ff8d8146070 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40099b0 tx=0x7ff8c4009250 comp rx=0 tx=0).stop 2023-09-30T00:47:59.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.866+0000 7ff8df01c700 1 -- 172.21.15.40:0/814499531 shutdown_connections 2023-09-30T00:47:59.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.866+0000 7ff8df01c700 1 --2- 172.21.15.40:0/814499531 >> v2:172.21.15.40:3300/0 conn(0x7ff8d80adfe0 0x7ff8d81465b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.866+0000 7ff8df01c700 1 --2- 172.21.15.40:0/814499531 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8d8068b10 0x7ff8d8146070 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.866+0000 7ff8df01c700 1 -- 172.21.15.40:0/814499531 >> 172.21.15.40:0/814499531 conn(0x7ff8d8067a00 msgr2=0x7ff8d80631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:59.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.866+0000 7ff8df01c700 1 -- 172.21.15.40:0/814499531 shutdown_connections 2023-09-30T00:47:59.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.866+0000 7ff8df01c700 1 -- 172.21.15.40:0/814499531 wait complete. 2023-09-30T00:47:59.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.867+0000 7ff8df01c700 10 monclient: build_initial_monmap 2023-09-30T00:47:59.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.867+0000 7ff8df01c700 10 monclient: monmap: 2023-09-30T00:47:59.870 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:59.870 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:59.870 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:47:59.870 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:47:59.870 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:59.870 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:47:59.870 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:47:59.871 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:47:59.871 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:47:59.871 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.868+0000 7ff8df01c700 1 librados: starting msgr at 2023-09-30T00:47:59.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.868+0000 7ff8df01c700 1 librados: starting objecter 2023-09-30T00:47:59.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.868+0000 7ff8df01c700 1 Processor -- start 2023-09-30T00:47:59.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.868+0000 7ff8df01c700 1 -- start start 2023-09-30T00:47:59.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.868+0000 7ff8df01c700 1 librados: setting wanted keys 2023-09-30T00:47:59.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.868+0000 7ff8df01c700 1 librados: calling monclient init 2023-09-30T00:47:59.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.868+0000 7ff8df01c700 10 monclient: init 2023-09-30T00:47:59.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.869+0000 7ff8df01c700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:59.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.869+0000 7ff8df01c700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:47:59.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.869+0000 7ff8df01c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8d8068b10 0x7ff8d80a9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.869+0000 7ff8df01c700 10 monclient(hunting): picked mon.noname-a con 0x7ff8d8068b10 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:47:59.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.869+0000 7ff8df01c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8d80adfe0 0x7ff8d80abce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.869+0000 7ff8df01c700 10 monclient(hunting): picked mon.noname-b con 0x7ff8d80adfe0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:47:59.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.869+0000 7ff8df01c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8d80af5f0 0x7ff8d80a9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.869+0000 7ff8df01c700 10 monclient(hunting): picked mon.noname-c con 0x7ff8d80af5f0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:47:59.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.869+0000 7ff8df01c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.869+0000 7ff8df01c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d80ac220 con 0x7ff8d8068b10 2023-09-30T00:47:59.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.869+0000 7ff8df01c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.869+0000 7ff8df01c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d80a71b0 con 0x7ff8d80adfe0 2023-09-30T00:47:59.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.870+0000 7ff8df01c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.870+0000 7ff8df01c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d80ac490 con 0x7ff8d80af5f0 2023-09-30T00:47:59.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.870+0000 7ff8df01c700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:59.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.870+0000 7ff8df01c700 10 monclient(hunting): authenticate will time out at 5341.171020s 2023-09-30T00:47:59.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.871+0000 7ff8dd5b9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8d80af5f0 0x7ff8d80a9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:59.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.871+0000 7ff8dd5b9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8d80af5f0 0x7ff8d80a9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:36754/0 (socket says 172.21.15.40:36754) 2023-09-30T00:47:59.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.871+0000 7ff8dd5b9700 1 -- 172.21.15.40:0/2132247764 learned_addr learned my addr 172.21.15.40:0/2132247764 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:47:59.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.871+0000 7ff8dd5b9700 10 monclient(hunting): get_auth_request con 0x7ff8d80af5f0 auth_method 0 2023-09-30T00:47:59.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.871+0000 7ff8dd5b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:59.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.871+0000 7ff8dd5b9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:59.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.871+0000 7ff8dd5b9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:59.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.872+0000 7ff8dd5b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:59.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.872+0000 7ff8dd5b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:59.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.872+0000 7ff8dd5b9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:59.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.877+0000 7ff8dd5b9700 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2023-09-30T00:47:59.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.877+0000 7ff8dd5b9700 1 -- 172.21.15.40:0/2132247764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8d80adfe0 msgr2=0x7ff8d80abce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.877+0000 7ff8dd5b9700 1 --2- 172.21.15.40:0/2132247764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8d80adfe0 0x7ff8d80abce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.877+0000 7ff8dd5b9700 1 -- 172.21.15.40:0/2132247764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8d8068b10 msgr2=0x7ff8d80a9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:59.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.877+0000 7ff8dd5b9700 1 --2- 172.21.15.40:0/2132247764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8d8068b10 0x7ff8d80a9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:59.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.877+0000 7ff8dd5b9700 10 monclient: _finish_hunting 0 2023-09-30T00:47:59.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.877+0000 7ff8dd5b9700 1 monclient: found mon.noname-c 2023-09-30T00:47:59.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.877+0000 7ff8dd5b9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:59.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.877+0000 7ff8dd5b9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:47:59.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.877+0000 7ff8dd5b9700 1 -- 172.21.15.40:0/2132247764 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c40094f0 con 0x7ff8d80af5f0 2023-09-30T00:47:59.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.880+0000 7ff8dd5b9700 1 --2- 172.21.15.40:0/2132247764 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8d80af5f0 0x7ff8d80a9c40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000bb60 tx=0x7ff8d0009810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:59.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.880+0000 7ff8d4ff9700 10 client.?.objecter ms_handle_connect 0x7ff8d80af5f0 2023-09-30T00:47:59.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.881+0000 7ff8d4ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:47:59.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8d4ff9700 1 -- 172.21.15.40:0/2132247764 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d000c940 con 0x7ff8d80af5f0 2023-09-30T00:47:59.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8d4ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:59.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8d4ff9700 10 monclient: dump: 2023-09-30T00:47:59.889 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.889 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.889 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.889 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.889 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.889 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.889 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.889 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.890 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.890 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8d4ff9700 10 monclient: _finish_auth 0 2023-09-30T00:47:59.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8d4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:59.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8d4ff9700 1 -- 172.21.15.40:0/2132247764 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8d000cac0 con 0x7ff8d80af5f0 2023-09-30T00:47:59.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8d4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:59.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8d4ff9700 1 -- 172.21.15.40:0/2132247764 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d000ce80 con 0x7ff8d80af5f0 2023-09-30T00:47:59.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:59.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8d4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:59.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8d4ff9700 10 monclient: dump: 2023-09-30T00:47:59.890 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:47:59.891 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.891 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.891 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:47:59.891 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:59.891 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.891 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:47:59.891 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:47:59.891 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:47:59.891 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8df01c700 5 monclient: authenticate success, global_id 4307 2023-09-30T00:47:59.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8df01c700 10 monclient: _renew_subs 2023-09-30T00:47:59.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8df01c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:59.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8d6ffd700 4 set_mon_vals no callback set 2023-09-30T00:47:59.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8df01c700 1 -- 172.21.15.40:0/2132247764 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d80ab8c0 con 0x7ff8d80af5f0 2023-09-30T00:47:59.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8df01c700 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:47:59.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8df01c700 10 monclient: _renew_subs 2023-09-30T00:47:59.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8df01c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:59.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8df01c700 1 -- 172.21.15.40:0/2132247764 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d80aa200 con 0x7ff8d80af5f0 2023-09-30T00:47:59.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.886+0000 7ff8df01c700 1 librados: init done 2023-09-30T00:47:59.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.887+0000 7ff8df01c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:59.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.887+0000 7ff8df01c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:59.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.887+0000 7ff8df01c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:47:59.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.887+0000 7ff8df01c700 1 -- 172.21.15.40:0/2132247764 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d80aba50 con 0x7ff8d80af5f0 2023-09-30T00:47:59.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.924+0000 7f1bbc9f8700 10 monclient: get_monmap_and_config 2023-09-30T00:47:59.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.924+0000 7f1bbc9f8700 10 monclient: build_initial_monmap 2023-09-30T00:47:59.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.924+0000 7f1bbc9f8700 10 monclient: monmap: 2023-09-30T00:47:59.925 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:47:59.925 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:47:59.925 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:47:59.925538+0000 2023-09-30T00:47:59.925 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:47:59.925538+0000 2023-09-30T00:47:59.926 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:59.926 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:47:59.926 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:47:59.926 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:47:59.926 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:47:59.926 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:47:59.926 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:47:59.926 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:47:59.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.925+0000 7f1bbc9f8700 1 Processor -- start 2023-09-30T00:47:59.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.925+0000 7f1bbc9f8700 1 -- start start 2023-09-30T00:47:59.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.925+0000 7f1bbc9f8700 10 monclient: init 2023-09-30T00:47:59.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.926+0000 7f1bbc9f8700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:59.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.926+0000 7f1bbc9f8700 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-09-30T00:47:59.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.926+0000 7f1bbc9f8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1bb80b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:47:59.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.926+0000 7f1bbc9f8700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1bb80b1830 0x7f1bb80a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:59.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.926+0000 7f1bbc9f8700 10 monclient(hunting): picked mon.noname-c con 0x7f1bb80b1830 addr v2:172.21.15.135:3300/0 2023-09-30T00:47:59.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.926+0000 7f1bbc9f8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bb813ac20 addr v1:172.21.15.135:6789/0 2023-09-30T00:47:59.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.926+0000 7f1bbc9f8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb80a5b20 con 0x7f1bb80b1460 2023-09-30T00:47:59.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.926+0000 7f1bbc9f8700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb80a71f0 con 0x7f1bb813ac20 2023-09-30T00:47:59.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.926+0000 7f1bbc9f8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:59.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.926+0000 7f1bbc9f8700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb8141e40 con 0x7f1bb80b1830 2023-09-30T00:47:59.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.927+0000 7f1bbc9f8700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:59.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:47:59.927+0000 7f1bbc9f8700 10 monclient(hunting): authenticate will time out at 5341.228014s 2023-09-30T00:47:59.978 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:48:00.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.001+0000 7ff8d4ff9700 1 -- 172.21.15.40:0/2132247764 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ff8d0014620 con 0x7ff8d80af5f0 2023-09-30T00:48:00.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.002+0000 7ff8d4ff9700 1 --2- 172.21.15.40:0/2132247764 >> v2:172.21.15.135:6800/116426 conn(0x7ff8bc0689e0 0x7ff8bc06ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.002+0000 7ff8d4ff9700 1 -- 172.21.15.40:0/2132247764 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff8d00847c0 con 0x7ff8d80af5f0 2023-09-30T00:48:00.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.002+0000 7ff8d4ff9700 10 client.4307.objecter ms_dispatch 0x7ff8d8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:00.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.002+0000 7ff8d4ff9700 3 client.4307.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:00.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.002+0000 7ff8d4ff9700 3 client.4307.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:00.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.002+0000 7ff8d4ff9700 20 client.4307.objecter dump_active .. 0 homeless 2023-09-30T00:48:00.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.002+0000 7ff8dcdb8700 1 --2- 172.21.15.40:0/2132247764 >> v2:172.21.15.135:6800/116426 conn(0x7ff8bc0689e0 0x7ff8bc06ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:00.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.002+0000 7ff8dcdb8700 10 monclient: get_auth_request con 0x7ff8bc0689e0 auth_method 0 2023-09-30T00:48:00.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.002+0000 7ff8d4ff9700 10 client.4307.objecter ms_handle_connect 0x7ff8bc0689e0 2023-09-30T00:48:00.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.002+0000 7ff8dcdb8700 1 --2- 172.21.15.40:0/2132247764 >> v2:172.21.15.135:6800/116426 conn(0x7ff8bc0689e0 0x7ff8bc06ae90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009720 tx=0x7ff8cc006c30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:00.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.015+0000 7fec3d371700 10 monclient: get_monmap_and_config 2023-09-30T00:48:00.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.015+0000 7fec3d371700 10 monclient: build_initial_monmap 2023-09-30T00:48:00.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.015+0000 7fec3d371700 10 monclient: monmap: 2023-09-30T00:48:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:00.016979+0000 2023-09-30T00:48:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:00.016979+0000 2023-09-30T00:48:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:00.018 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:00.018 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.016+0000 7fec3d371700 1 Processor -- start 2023-09-30T00:48:00.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.017+0000 7fec3d371700 1 -- start start 2023-09-30T00:48:00.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.017+0000 7fec3d371700 10 monclient: init 2023-09-30T00:48:00.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.018+0000 7fec3d371700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:00.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.018+0000 7fec3d371700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-09-30T00:48:00.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.018+0000 7fec3d371700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec380af460 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:00.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.018+0000 7fec3d371700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec38145bf0 addr v1:172.21.15.40:6789/0 2023-09-30T00:48:00.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.018+0000 7fec3d371700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3806ea40 0x7fec3814e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.018+0000 7fec3d371700 10 monclient(hunting): picked mon.noname-b con 0x7fec3806ea40 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:00.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.018+0000 7fec3d371700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3806e320 con 0x7fec38145bf0 2023-09-30T00:48:00.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.018+0000 7fec3d371700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec380ad540 con 0x7fec380af460 2023-09-30T00:48:00.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.018+0000 7fec3d371700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.018+0000 7fec3d371700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec38146050 con 0x7fec3806ea40 2023-09-30T00:48:00.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.018+0000 7fec3d371700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:00.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.018+0000 7fec3d371700 10 monclient(hunting): authenticate will time out at 5341.319237s 2023-09-30T00:48:00.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.019+0000 7f530fa15700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T00:48:00.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.019+0000 7f530fa15700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T00:48:00.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.019+0000 7f530fa15700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:00.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.019+0000 7f530fa15700 1 -- 172.21.15.40:0/788429292 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f53080a65d0 con 0x7f53080ae830 2023-09-30T00:48:00.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.023+0000 7fe9757fa700 1 -- 172.21.15.40:0/1369260279 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe970051930 con 0x7fe9780ae270 2023-09-30T00:48:00.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.024+0000 7fe9757fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:00.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.024+0000 7fe9757fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:00.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.048+0000 7f1bb659c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1bb80b1460 0x7f1bb8141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43424/0 (socket says 172.21.15.40:43424) 2023-09-30T00:48:00.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.048+0000 7f1bb659c700 1 -- 172.21.15.40:0/2596609527 learned_addr learned my addr 172.21.15.40:0/2596609527 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:48:00.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.060+0000 7fec367fc700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fec38145bf0 0x7fec3806dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43438/0 (socket says 172.21.15.40:43438) 2023-09-30T00:48:00.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.060+0000 7fec367fc700 1 -- 172.21.15.40:0/3291418337 learned_addr learned my addr 172.21.15.40:0/3291418337 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:48:00.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.070+0000 7f1bb559a700 1 -- 172.21.15.40:0/2596609527 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897680102 0 0) 0x7f1bb80a5b20 con 0x7f1bb80b1460 2023-09-30T00:48:00.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.070+0000 7f1bb559a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:00.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.070+0000 7f1bb559a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:00.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.070+0000 7f1bb559a700 10 monclient(hunting): my global_id is 4335 2023-09-30T00:48:00.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.070+0000 7f1bb559a700 1 -- 172.21.15.40:0/2596609527 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba0003570 con 0x7f1bb80b1460 2023-09-30T00:48:00.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.070+0000 7fec35ffb700 1 -- 172.21.15.40:0/3291418337 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134662599 0 0) 0x7fec3806e320 con 0x7fec38145bf0 2023-09-30T00:48:00.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.070+0000 7fec35ffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:00.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.070+0000 7fec35ffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:00.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.070+0000 7fec35ffb700 10 monclient(hunting): my global_id is 4338 2023-09-30T00:48:00.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.070+0000 7fec35ffb700 1 -- 172.21.15.40:0/3291418337 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec20003570 con 0x7fec38145bf0 2023-09-30T00:48:00.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 -- 172.21.15.40:0/2596609527 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1877777618 0 0) 0x7f1ba0003570 con 0x7f1bb80b1460 2023-09-30T00:48:00.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 -- 172.21.15.40:0/2596609527 >> v2:172.21.15.135:3300/0 conn(0x7f1bb80b1830 msgr2=0x7f1bb80a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:00.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 --2- 172.21.15.40:0/2596609527 >> v2:172.21.15.135:3300/0 conn(0x7f1bb80b1830 0x7f1bb80a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 -- 172.21.15.40:0/2596609527 >> v1:172.21.15.135:6789/0 conn(0x7f1bb813ac20 legacy=0x7f1bb813b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:00.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient: _finish_hunting 0 2023-09-30T00:48:00.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 monclient: found mon.noname-a-legacy 2023-09-30T00:48:00.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:00.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2023-09-30T00:48:00.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 -- 172.21.15.40:0/2596609527 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb80a74d0 con 0x7f1bb80b1460 2023-09-30T00:48:00.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient: _finish_auth 0 2023-09-30T00:48:00.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:00.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 -- 172.21.15.40:0/2596609527 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3033000540 0 0) 0x7f1bac001470 con 0x7f1bb80b1460 2023-09-30T00:48:00.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:00.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T00:48:00.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient: dump: 2023-09-30T00:48:00.084 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:00.084 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.084 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.084 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.084 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:00.084 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.084 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:00.084 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:00.084 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:00.084 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2023-09-30T00:48:00.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:00.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 -- 172.21.15.40:0/2596609527 >> v1:172.21.15.40:6789/0 conn(0x7f1bb80b1460 legacy=0x7f1bb8141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:48:00.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 --2- 172.21.15.40:0/2596609527 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ba0003a60 0x7f1ba0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient(hunting): picked mon.b con 0x7f1ba0003a60 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:00.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 --2- 172.21.15.40:0/2596609527 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1ba0004810 0x7f1ba0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient(hunting): picked mon.c con 0x7f1ba0004810 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:00.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 --2- 172.21.15.40:0/2596609527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba00052c0 0x7f1ba000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient(hunting): picked mon.a con 0x7f1ba00052c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:00.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 -- 172.21.15.40:0/2596609527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb8141e40 con 0x7f1ba00052c0 2023-09-30T00:48:00.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 -- 172.21.15.40:0/2596609527 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba000c280 con 0x7f1ba0003a60 2023-09-30T00:48:00.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 1 -- 172.21.15.40:0/2596609527 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba000c440 con 0x7f1ba0004810 2023-09-30T00:48:00.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:00.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.081+0000 7f1bb559a700 10 monclient(hunting): _finish_auth 0 2023-09-30T00:48:00.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.083+0000 7f1bb6d9d700 1 --2- 172.21.15.40:0/2596609527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba00052c0 0x7f1ba000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:00.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.084+0000 7f1bb6d9d700 10 monclient(hunting): get_auth_request con 0x7f1ba00052c0 auth_method 0 2023-09-30T00:48:00.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.084+0000 7f1bb6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:00.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.084+0000 7f1bb6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:00.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.084+0000 7f1bb6d9d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:48:00.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.084+0000 7f1bb6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:00.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.084+0000 7f1bb6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:00.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.084+0000 7f1bb6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:48:00.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.086+0000 7f1bb6d9d700 10 monclient(hunting): handle_auth_done global_id 4335 payload 293 2023-09-30T00:48:00.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.086+0000 7f1bb6d9d700 1 -- 172.21.15.40:0/2596609527 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1ba0004810 msgr2=0x7f1ba0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:00.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.086+0000 7f1bb6d9d700 1 --2- 172.21.15.40:0/2596609527 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1ba0004810 0x7f1ba0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.086+0000 7f1bb6d9d700 1 -- 172.21.15.40:0/2596609527 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ba0003a60 msgr2=0x7f1ba0003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:00.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.086+0000 7f1bb6d9d700 1 --2- 172.21.15.40:0/2596609527 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1ba0003a60 0x7f1ba0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.086+0000 7f1bb6d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:48:00.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.086+0000 7f1bb6d9d700 1 monclient: found mon.a 2023-09-30T00:48:00.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.086+0000 7f1bb6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:00.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.086+0000 7f1bb6d9d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:00.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.086+0000 7f1bb6d9d700 1 -- 172.21.15.40:0/2596609527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb80a74d0 con 0x7f1ba00052c0 2023-09-30T00:48:00.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.086+0000 7f1bb6d9d700 10 monclient: _finish_auth 0 2023-09-30T00:48:00.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.086+0000 7f1bb6d9d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:00.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.087+0000 7f1bb6d9d700 1 -- 172.21.15.40:0/2596609527 reap_dead start 2023-09-30T00:48:00.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.087+0000 7f1bbc9f8700 5 monclient: authenticate success, global_id 4335 2023-09-30T00:48:00.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.087+0000 7f1bbc9f8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:00.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 -- 172.21.15.40:0/3291418337 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 473597608 0 0) 0x7fec20003570 con 0x7fec38145bf0 2023-09-30T00:48:00.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 -- 172.21.15.40:0/3291418337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3806ea40 msgr2=0x7fec3814e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:00.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 --2- 172.21.15.40:0/3291418337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec3806ea40 0x7fec3814e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 -- 172.21.15.40:0/3291418337 >> v1:172.21.15.135:6789/0 conn(0x7fec380af460 legacy=0x7fec3806d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:00.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient: _finish_hunting 0 2023-09-30T00:48:00.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 monclient: found mon.noname-a-legacy 2023-09-30T00:48:00.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:00.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2023-09-30T00:48:00.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 -- 172.21.15.40:0/3291418337 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec380ae980 con 0x7fec38145bf0 2023-09-30T00:48:00.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient: _finish_auth 0 2023-09-30T00:48:00.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:00.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 -- 172.21.15.40:0/3291418337 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3033000540 0 0) 0x7fec28001470 con 0x7fec38145bf0 2023-09-30T00:48:00.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:00.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T00:48:00.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient: dump: 2023-09-30T00:48:00.092 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:00.092 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.092 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.092 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.092 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:00.092 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.092 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:00.092 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:00.093 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:00.093 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2023-09-30T00:48:00.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:00.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 -- 172.21.15.40:0/3291418337 >> v1:172.21.15.40:6789/0 conn(0x7fec38145bf0 legacy=0x7fec3806dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:48:00.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 --2- 172.21.15.40:0/3291418337 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec20003b10 0x7fec20003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient(hunting): picked mon.b con 0x7fec20003b10 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:00.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 --2- 172.21.15.40:0/3291418337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec20004810 0x7fec20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient(hunting): picked mon.a con 0x7fec20004810 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:00.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 --2- 172.21.15.40:0/3291418337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec20005410 0x7fec2000bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient(hunting): picked mon.c con 0x7fec20005410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:00.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 -- 172.21.15.40:0/3291418337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2000c320 con 0x7fec20004810 2023-09-30T00:48:00.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 -- 172.21.15.40:0/3291418337 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2000c490 con 0x7fec20003b10 2023-09-30T00:48:00.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 1 -- 172.21.15.40:0/3291418337 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2000c650 con 0x7fec20005410 2023-09-30T00:48:00.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:00.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.089+0000 7fec35ffb700 10 monclient(hunting): _finish_auth 0 2023-09-30T00:48:00.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.093+0000 7f1bb6d9d700 1 --2- 172.21.15.40:0/2596609527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba00052c0 0x7f1ba000ba60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0001ec0 tx=0x7f1bb00014d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67f02f23f3cc0a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:00.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.093+0000 7f22d67fc700 1 -- 172.21.15.40:0/4129146734 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f22d0008040 con 0x7f22e0155d90 2023-09-30T00:48:00.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.094+0000 7f22d67fc700 1 --2- 172.21.15.40:0/4129146734 >> v2:172.21.15.135:6800/116426 conn(0x7f22c4068960 0x7f22c406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.094+0000 7f22d67fc700 1 -- 172.21.15.40:0/4129146734 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f22d0083c80 con 0x7f22e0155d90 2023-09-30T00:48:00.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.094+0000 7f22d67fc700 10 client.4363.objecter ms_dispatch 0x7f22e0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:00.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.094+0000 7f22d67fc700 3 client.4363.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:00.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.094+0000 7f22d67fc700 3 client.4363.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:00.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.094+0000 7f22e639a700 1 --2- 172.21.15.40:0/4129146734 >> v2:172.21.15.135:6800/116426 conn(0x7f22c4068960 0x7f22c406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:00.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.094+0000 7f22e639a700 10 monclient: get_auth_request con 0x7f22c4068960 auth_method 0 2023-09-30T00:48:00.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.094+0000 7f22d67fc700 20 client.4363.objecter dump_active .. 0 homeless 2023-09-30T00:48:00.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.095+0000 7f22d67fc700 10 client.4363.objecter ms_handle_connect 0x7f22c4068960 2023-09-30T00:48:00.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.095+0000 7f22e639a700 1 --2- 172.21.15.40:0/4129146734 >> v2:172.21.15.135:6800/116426 conn(0x7f22c4068960 0x7f22c406ae10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f22d80092b0 tx=0x7f22d8017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:00.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.096+0000 7fec367fc700 1 --2- 172.21.15.40:0/3291418337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec20004810 0x7fec20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:00.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.100+0000 7f1bb559a700 1 -- 172.21.15.40:0/2596609527 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0001800 con 0x7f1ba00052c0 2023-09-30T00:48:00.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.100+0000 7f1bb559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:00.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.100+0000 7f1bb559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:00.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.100+0000 7f1bb559a700 10 monclient: dump: 2023-09-30T00:48:00.102 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:00.102 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.102 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.102 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.102 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:00.103 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.103 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:00.103 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:00.103 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:00.103 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.100+0000 7f1bb559a700 1 -- 172.21.15.40:0/2596609527 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb000c6a0 con 0x7f1ba00052c0 2023-09-30T00:48:00.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.100+0000 7f1bb559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:00.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.100+0000 7f1bb559a700 1 -- 172.21.15.40:0/2596609527 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb000ca60 con 0x7f1ba00052c0 2023-09-30T00:48:00.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.100+0000 7f1bb559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:00.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.100+0000 7f1bb559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:00.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.100+0000 7f1bb559a700 10 monclient: dump: 2023-09-30T00:48:00.104 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:00.105 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.105 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.105 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.105 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:00.105 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.105 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:00.106 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:00.106 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:00.106 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.100+0000 7f1bbc9f8700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:00.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.100+0000 7f1bbc9f8700 4 set_mon_vals no callback set 2023-09-30T00:48:00.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.101+0000 7fec367fc700 10 monclient(hunting): get_auth_request con 0x7fec20004810 auth_method 0 2023-09-30T00:48:00.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.101+0000 7fec367fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:00.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.101+0000 7fec367fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:00.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.101+0000 7fec367fc700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:48:00.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.101+0000 7fec367fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:00.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.101+0000 7fec367fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:00.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.101+0000 7fec367fc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:48:00.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.101+0000 7f1bbc9f8700 10 monclient: shutdown 2023-09-30T00:48:00.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.101+0000 7f1bbc9f8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:00.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.101+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/2596609527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba00052c0 msgr2=0x7f1ba000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.101+0000 7f1bbc9f8700 1 --2- 172.21.15.40:0/2596609527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba00052c0 0x7f1ba000ba60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0001ec0 tx=0x7f1bb00014d0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.102+0000 7f22d67fc700 1 -- 172.21.15.40:0/4129146734 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f22d0051620 con 0x7f22e0155d90 2023-09-30T00:48:00.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.102+0000 7f22d67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:00.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.102+0000 7f22d67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:00.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.103+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/2596609527 shutdown_connections 2023-09-30T00:48:00.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.103+0000 7f1bbc9f8700 1 --2- 172.21.15.40:0/2596609527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba00052c0 0x7f1ba000ba60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.103+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/2596609527 >> 172.21.15.40:0/2596609527 conn(0x7f1bb80631f0 msgr2=0x7f1bb80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:00.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.104+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/2596609527 shutdown_connections 2023-09-30T00:48:00.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.104+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/2596609527 wait complete. 2023-09-30T00:48:00.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.105+0000 7f1bbc9f8700 10 monclient: build_initial_monmap 2023-09-30T00:48:00.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.105+0000 7f1bbc9f8700 10 monclient: monmap: 2023-09-30T00:48:00.112 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:00.112 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:00.113 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:00.113 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:00.113 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:00.113 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:00.113 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:00.113 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:00.114 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:00.114 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.105+0000 7f1bbc9f8700 1 librados: starting msgr at 2023-09-30T00:48:00.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.105+0000 7f1bbc9f8700 1 librados: starting objecter 2023-09-30T00:48:00.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.106+0000 7f1bbc9f8700 1 Processor -- start 2023-09-30T00:48:00.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.106+0000 7f1bbc9f8700 1 -- start start 2023-09-30T00:48:00.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.106+0000 7f1bbc9f8700 1 librados: setting wanted keys 2023-09-30T00:48:00.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.106+0000 7f1bbc9f8700 1 librados: calling monclient init 2023-09-30T00:48:00.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.106+0000 7f1bbc9f8700 10 monclient: init 2023-09-30T00:48:00.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:00.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:48:00.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba00052c0 0x7f1bb813dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 10 monclient(hunting): picked mon.noname-a con 0x7f1ba00052c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:00.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bb806c860 0x7f1bb806d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 10 monclient(hunting): picked mon.noname-c con 0x7f1bb806c860 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:00.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bb8151fe0 0x7f1bb8152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 10 monclient(hunting): picked mon.noname-b con 0x7f1bb8151fe0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:00.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb806cdd0 con 0x7f1ba00052c0 2023-09-30T00:48:00.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb806e6a0 con 0x7f1bb8151fe0 2023-09-30T00:48:00.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb813ec20 con 0x7f1bb806c860 2023-09-30T00:48:00.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:00.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.107+0000 7f1bbc9f8700 10 monclient(hunting): authenticate will time out at 5341.408310s 2023-09-30T00:48:00.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.108+0000 7f1bb659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba00052c0 0x7f1bb813dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:00.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.108+0000 7f1bb659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba00052c0 0x7f1bb813dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56808/0 (socket says 172.21.15.40:56808) 2023-09-30T00:48:00.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.108+0000 7f1bb659c700 1 -- 172.21.15.40:0/4239756726 learned_addr learned my addr 172.21.15.40:0/4239756726 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:00.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.108+0000 7f1bb659c700 10 monclient(hunting): get_auth_request con 0x7f1ba00052c0 auth_method 0 2023-09-30T00:48:00.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.108+0000 7f1bb659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:00.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.108+0000 7f1bb659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:00.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.108+0000 7f1bb659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:00.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.109+0000 7f1bb659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:00.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.109+0000 7f1bb659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:00.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.109+0000 7f1bb659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:00.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.113+0000 7f1bb659c700 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2023-09-30T00:48:00.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.113+0000 7f1bb659c700 1 -- 172.21.15.40:0/4239756726 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bb806c860 msgr2=0x7f1bb806d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.113+0000 7f1bb659c700 1 --2- 172.21.15.40:0/4239756726 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bb806c860 0x7f1bb806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.113+0000 7f1bb659c700 1 -- 172.21.15.40:0/4239756726 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bb8151fe0 msgr2=0x7f1bb8152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.113+0000 7f1bb659c700 1 --2- 172.21.15.40:0/4239756726 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bb8151fe0 0x7f1bb8152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.113+0000 7f1bb659c700 10 monclient: _finish_hunting 0 2023-09-30T00:48:00.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.113+0000 7f1bb659c700 1 monclient: found mon.noname-a 2023-09-30T00:48:00.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.113+0000 7f1bb659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:00.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.113+0000 7f1bb659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:00.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.113+0000 7f1bb659c700 1 -- 172.21.15.40:0/4239756726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb806bfd0 con 0x7f1ba00052c0 2023-09-30T00:48:00.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.114+0000 7fec367fc700 10 monclient(hunting): handle_auth_done global_id 4338 payload 293 2023-09-30T00:48:00.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.114+0000 7fec367fc700 1 -- 172.21.15.40:0/3291418337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec20005410 msgr2=0x7fec2000bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:00.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.114+0000 7fec367fc700 1 --2- 172.21.15.40:0/3291418337 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec20005410 0x7fec2000bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.114+0000 7fec367fc700 1 -- 172.21.15.40:0/3291418337 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec20003b10 msgr2=0x7fec20003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:00.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.114+0000 7fec367fc700 1 --2- 172.21.15.40:0/3291418337 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec20003b10 0x7fec20003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.114+0000 7fec367fc700 10 monclient: _finish_hunting 0 2023-09-30T00:48:00.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.114+0000 7fec367fc700 1 monclient: found mon.a 2023-09-30T00:48:00.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.114+0000 7fec367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:00.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.114+0000 7fec367fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:00.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.114+0000 7fec367fc700 1 -- 172.21.15.40:0/3291418337 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec380ae980 con 0x7fec20004810 2023-09-30T00:48:00.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.114+0000 7fec367fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:00.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.114+0000 7fec367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:00.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.117+0000 7fec367fc700 1 --2- 172.21.15.40:0/3291418337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec20004810 0x7fec20004c80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fec2800c3e0 tx=0x7fec28001e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1541444d8ad1be5b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:00.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.117+0000 7f52fd7fa700 1 -- 172.21.15.40:0/788429292 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5300021090 con 0x7f53080ae830 2023-09-30T00:48:00.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.117+0000 7f52fd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T00:48:00.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.117+0000 7f52fd7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:00.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.117+0000 7fec3d371700 5 monclient: authenticate success, global_id 4338 2023-09-30T00:48:00.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.117+0000 7fec3d371700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:00.122 INFO:teuthology.orchestra.run.smithi040.stdout:64424509445 2023-09-30T00:48:00.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.117+0000 7f1bb659c700 1 --2- 172.21.15.40:0/4239756726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba00052c0 0x7f1bb813dfa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1bac00c930 tx=0x7f1bac001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:00.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.117+0000 7f1ba67fc700 10 client.?.objecter ms_handle_connect 0x7f1ba00052c0 2023-09-30T00:48:00.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.117+0000 7f1ba67fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:00.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.122+0000 7fec35ffb700 1 -- 172.21.15.40:0/3291418337 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2800ccb0 con 0x7fec20004810 2023-09-30T00:48:00.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.122+0000 7fec35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:00.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.122+0000 7fec35ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:00.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.122+0000 7fec35ffb700 10 monclient: dump: 2023-09-30T00:48:00.123 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:00.123 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.124 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.124 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.124 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:00.124 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.124 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:00.124 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:00.124 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:00.124 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.120+0000 7f530fa15700 10 librados: watch_flush enter 2023-09-30T00:48:00.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.120+0000 7f530fa15700 10 librados: watch_flush exit 2023-09-30T00:48:00.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.120+0000 7f530fa15700 20 client.4357.objecter shutdown clearing up homeless session... 2023-09-30T00:48:00.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.120+0000 7f530fa15700 10 client.4357.objecter successfully canceled tick 2023-09-30T00:48:00.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.121+0000 7f530fa15700 1 -- 172.21.15.40:0/788429292 >> v2:172.21.15.135:6800/116426 conn(0x7f52ec068890 msgr2=0x7f52ec06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.121+0000 7f530fa15700 1 --2- 172.21.15.40:0/788429292 >> v2:172.21.15.135:6800/116426 conn(0x7f52ec068890 0x7f52ec06ad40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5304006fd0 tx=0x7f5304008040 comp rx=0 tx=0).stop 2023-09-30T00:48:00.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.121+0000 7f530fa15700 10 monclient: shutdown 2023-09-30T00:48:00.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.121+0000 7f530fa15700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:00.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.121+0000 7f530fa15700 1 -- 172.21.15.40:0/788429292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53080ae830 msgr2=0x7f530811f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.121+0000 7f530fa15700 1 --2- 172.21.15.40:0/788429292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53080ae830 0x7f530811f650 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5300015000 tx=0x7f5300018000 comp rx=0 tx=0).stop 2023-09-30T00:48:00.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.121+0000 7f530fa15700 1 -- 172.21.15.40:0/788429292 shutdown_connections 2023-09-30T00:48:00.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.121+0000 7f530fa15700 1 --2- 172.21.15.40:0/788429292 >> v2:172.21.15.135:6800/116426 conn(0x7f52ec068890 0x7f52ec06ad40 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5304006fd0 tx=0x7f5304008040 comp rx=0 tx=0).stop 2023-09-30T00:48:00.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.121+0000 7f530fa15700 1 --2- 172.21.15.40:0/788429292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f53080ae830 0x7f530811f650 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.121+0000 7f530fa15700 1 --2- 172.21.15.40:0/788429292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5308067d30 0x7f5308122d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.121+0000 7f530fa15700 1 --2- 172.21.15.40:0/788429292 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f53080ad4e0 0x7f5308121fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.121+0000 7f530fa15700 1 -- 172.21.15.40:0/788429292 >> 172.21.15.40:0/788429292 conn(0x7f5308068920 msgr2=0x7f53080afe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:00.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.126+0000 7fec35ffb700 1 -- 172.21.15.40:0/3291418337 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2800ce10 con 0x7fec20004810 2023-09-30T00:48:00.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.126+0000 7fec35ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:00.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.126+0000 7fec35ffb700 1 -- 172.21.15.40:0/3291418337 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec28005b50 con 0x7fec20004810 2023-09-30T00:48:00.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.126+0000 7fec35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:00.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.126+0000 7fec35ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:00.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.126+0000 7fec35ffb700 10 monclient: dump: 2023-09-30T00:48:00.128 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:00.129 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.129 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.129 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.129 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:00.129 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.129 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:00.129 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:00.129 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:00.129 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1ba67fc700 1 -- 172.21.15.40:0/4239756726 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac007360 con 0x7f1ba00052c0 2023-09-30T00:48:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1ba67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1ba67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1ba67fc700 10 monclient: dump: 2023-09-30T00:48:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:00.131 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1ba67fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1ba67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1ba67fc700 1 -- 172.21.15.40:0/4239756726 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bac0074c0 con 0x7f1ba00052c0 2023-09-30T00:48:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1ba67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1ba67fc700 1 -- 172.21.15.40:0/4239756726 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bac004830 con 0x7f1ba00052c0 2023-09-30T00:48:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1ba67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1ba67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1ba67fc700 10 monclient: dump: 2023-09-30T00:48:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:00.132 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:00.133 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.128+0000 7f530fa15700 1 -- 172.21.15.40:0/788429292 shutdown_connections 2023-09-30T00:48:00.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7fec3d371700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:00.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7fec3d371700 4 set_mon_vals no callback set 2023-09-30T00:48:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.128+0000 7fec3d371700 10 monclient: shutdown 2023-09-30T00:48:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.128+0000 7fec3d371700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.128+0000 7fec3d371700 1 -- 172.21.15.40:0/3291418337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec20004810 msgr2=0x7fec20004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.128+0000 7fec3d371700 1 --2- 172.21.15.40:0/3291418337 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec20004810 0x7fec20004c80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fec2800c3e0 tx=0x7fec28001e60 comp rx=0 tx=0).stop 2023-09-30T00:48:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.128+0000 7fec377fe700 1 -- 172.21.15.40:0/3291418337 reap_dead start 2023-09-30T00:48:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1bbc9f8700 5 monclient: authenticate success, global_id 4341 2023-09-30T00:48:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1bbc9f8700 10 monclient: _renew_subs 2023-09-30T00:48:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1bbc9f8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.127+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/4239756726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb8155510 con 0x7f1ba00052c0 2023-09-30T00:48:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.128+0000 7f1bbc9f8700 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.128+0000 7f1bbc9f8700 10 monclient: _renew_subs 2023-09-30T00:48:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.128+0000 7f1bbc9f8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.128+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/4239756726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb81559b0 con 0x7f1ba00052c0 2023-09-30T00:48:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.128+0000 7f1bbc9f8700 1 librados: init done 2023-09-30T00:48:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.128+0000 7fec3d371700 1 -- 172.21.15.40:0/3291418337 shutdown_connections 2023-09-30T00:48:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.128+0000 7fec3d371700 1 -- 172.21.15.40:0/3291418337 >> 172.21.15.40:0/3291418337 conn(0x7fec3806af40 msgr2=0x7fec380631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.130+0000 7f1bb4d99700 4 set_mon_vals no callback set 2023-09-30T00:48:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.131+0000 7f1bbc9f8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.131+0000 7f1bbc9f8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.131+0000 7f1bbc9f8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.131+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/4239756726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b9c0050b0 con 0x7f1ba00052c0 2023-09-30T00:48:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.133+0000 7f530fa15700 1 -- 172.21.15.40:0/788429292 wait complete. 2023-09-30T00:48:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.133+0000 7f530fa15700 1 librados: shutdown 2023-09-30T00:48:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.133+0000 7fec3d371700 1 -- 172.21.15.40:0/3291418337 shutdown_connections 2023-09-30T00:48:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.133+0000 7fec3d371700 1 -- 172.21.15.40:0/3291418337 wait complete. 2023-09-30T00:48:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.134+0000 7fec3d371700 10 monclient: build_initial_monmap 2023-09-30T00:48:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.134+0000 7fec3d371700 10 monclient: monmap: 2023-09-30T00:48:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:00.138 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:00.138 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:00.138 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:00.138 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:00.138 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:00.138 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.135+0000 7fec3d371700 1 librados: starting msgr at 2023-09-30T00:48:00.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.135+0000 7fec3d371700 1 librados: starting objecter 2023-09-30T00:48:00.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.135+0000 7fec3d371700 1 Processor -- start 2023-09-30T00:48:00.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.137+0000 7fec3d371700 1 -- start start 2023-09-30T00:48:00.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.137+0000 7fec3d371700 1 librados: setting wanted keys 2023-09-30T00:48:00.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.137+0000 7fec3d371700 1 librados: calling monclient init 2023-09-30T00:48:00.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.137+0000 7fec3d371700 10 monclient: init 2023-09-30T00:48:00.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.137+0000 7fec3d371700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:00.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:48:00.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec20004810 0x7fec380b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 10 monclient(hunting): picked mon.noname-a con 0x7fec20004810 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:00.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec380a54c0 0x7fec380ac650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 10 monclient(hunting): picked mon.noname-c con 0x7fec380a54c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:00.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec3815a560 0x7fec3815a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 10 monclient(hunting): picked mon.noname-b con 0x7fec3815a560 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:00.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec380acb90 con 0x7fec20004810 2023-09-30T00:48:00.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec380a5b00 con 0x7fec3815a560 2023-09-30T00:48:00.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3815af20 con 0x7fec380a54c0 2023-09-30T00:48:00.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:00.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.138+0000 7fec3d371700 10 monclient(hunting): authenticate will time out at 5341.439128s 2023-09-30T00:48:00.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.139+0000 7fec377fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec3815a560 0x7fec3815a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:00.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.139+0000 7fec377fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec3815a560 0x7fec3815a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43504/0 (socket says 172.21.15.40:43504) 2023-09-30T00:48:00.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.139+0000 7fec377fe700 1 -- 172.21.15.40:0/1446213468 learned_addr learned my addr 172.21.15.40:0/1446213468 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:00.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.139+0000 7fec377fe700 10 monclient(hunting): get_auth_request con 0x7fec3815a560 auth_method 0 2023-09-30T00:48:00.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.139+0000 7fec377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:00.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.139+0000 7fec377fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:00.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.139+0000 7fec377fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:00.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.140+0000 7fec377fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:00.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.140+0000 7fec377fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:00.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.140+0000 7fec377fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:00.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.143+0000 7fec377fe700 10 monclient(hunting): handle_auth_done global_id 4366 payload 824 2023-09-30T00:48:00.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.143+0000 7fec377fe700 1 -- 172.21.15.40:0/1446213468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec380a54c0 msgr2=0x7fec380ac650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.143+0000 7fec377fe700 1 --2- 172.21.15.40:0/1446213468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec380a54c0 0x7fec380ac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.143+0000 7fec377fe700 1 -- 172.21.15.40:0/1446213468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec20004810 msgr2=0x7fec380b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.143+0000 7fec377fe700 1 --2- 172.21.15.40:0/1446213468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec20004810 0x7fec380b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.143+0000 7fec377fe700 10 monclient: _finish_hunting 0 2023-09-30T00:48:00.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.143+0000 7fec377fe700 1 monclient: found mon.noname-b 2023-09-30T00:48:00.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.143+0000 7fec377fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:00.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.143+0000 7fec377fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:00.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.143+0000 7fec377fe700 1 -- 172.21.15.40:0/1446213468 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3815b090 con 0x7fec3815a560 2023-09-30T00:48:00.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.146+0000 7fec377fe700 1 --2- 172.21.15.40:0/1446213468 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec3815a560 0x7fec3815a9e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fec380af150 tx=0x7fec30003030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:00.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.146+0000 7fec1effd700 10 client.?.objecter ms_handle_connect 0x7fec3815a560 2023-09-30T00:48:00.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.146+0000 7fec1effd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:00.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.156+0000 7f1d7e6e7700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:48:00.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.156+0000 7f1d7e6e7700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:48:00.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.156+0000 7f1d7e6e7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:48:00.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.156+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/3436642936 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1d78064a00 con 0x7f1d780af410 2023-09-30T00:48:00.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec1effd700 1 -- 172.21.15.40:0/1446213468 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30007360 con 0x7fec3815a560 2023-09-30T00:48:00.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:00.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec1effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:00.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec1effd700 10 monclient: dump: 2023-09-30T00:48:00.162 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:00.162 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.162 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.163 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.163 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:00.163 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.163 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:00.163 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:00.163 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:00.163 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec1effd700 10 monclient: _finish_auth 0 2023-09-30T00:48:00.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec1effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:00.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec3d371700 5 monclient: authenticate success, global_id 4366 2023-09-30T00:48:00.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec1effd700 1 -- 172.21.15.40:0/1446213468 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec300045d0 con 0x7fec3815a560 2023-09-30T00:48:00.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec1effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:00.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec1effd700 1 -- 172.21.15.40:0/1446213468 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec300074e0 con 0x7fec3815a560 2023-09-30T00:48:00.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec3d371700 10 monclient: _renew_subs 2023-09-30T00:48:00.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec3d371700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:00.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec3d371700 1 -- 172.21.15.40:0/1446213468 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec380aa920 con 0x7fec3815a560 2023-09-30T00:48:00.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec357fa700 4 set_mon_vals no callback set 2023-09-30T00:48:00.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:00.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec1effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:00.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec3d371700 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:00.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec1effd700 10 monclient: dump: 2023-09-30T00:48:00.164 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:00.164 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:00.165 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec3d371700 10 monclient: _renew_subs 2023-09-30T00:48:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec3d371700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec3d371700 1 -- 172.21.15.40:0/1446213468 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec380aadc0 con 0x7fec3815a560 2023-09-30T00:48:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.161+0000 7fec3d371700 1 librados: init done 2023-09-30T00:48:00.171 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.7 2023-09-30T00:48:00.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.175+0000 7fec3d371700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:00.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.176+0000 7fec3d371700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:00.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.176+0000 7fec3d371700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.176+0000 7fec3d371700 1 -- 172.21.15.40:0/1446213468 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec380b0110 con 0x7fec3815a560 2023-09-30T00:48:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.176+0000 7fa0aef9a700 10 monclient: get_monmap_and_config 2023-09-30T00:48:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.176+0000 7fa0aef9a700 10 monclient: build_initial_monmap 2023-09-30T00:48:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.176+0000 7fa0aef9a700 10 monclient: monmap: 2023-09-30T00:48:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:00.177189+0000 2023-09-30T00:48:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:00.177189+0000 2023-09-30T00:48:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.178 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:00.178 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:00.178 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:00.178 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:00.178 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:00.178 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.177+0000 7fa0aef9a700 1 Processor -- start 2023-09-30T00:48:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.177+0000 7fa0aef9a700 1 -- start start 2023-09-30T00:48:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.177+0000 7fa0aef9a700 10 monclient: init 2023-09-30T00:48:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.178+0000 7fa0aef9a700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.178+0000 7fa0aef9a700 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-09-30T00:48:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.178+0000 7fa0aef9a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa0a80b14a0 0x7fa0a80afe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.178+0000 7fa0aef9a700 10 monclient(hunting): picked mon.noname-a con 0x7fa0a80b14a0 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.178+0000 7fa0aef9a700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa0a80b1870 0x7fa0a8145b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.178+0000 7fa0aef9a700 10 monclient(hunting): picked mon.noname-c con 0x7fa0a80b1870 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.178+0000 7fa0aef9a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa0a81460b0 addr v1:172.21.15.40:6789/0 2023-09-30T00:48:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.178+0000 7fa0aef9a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a813ff40 con 0x7fa0a81460b0 2023-09-30T00:48:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.178+0000 7fa0aef9a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.178+0000 7fa0aef9a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a806b030 con 0x7fa0a80b14a0 2023-09-30T00:48:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.178+0000 7fa0aef9a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.178+0000 7fa0aef9a700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a806b310 con 0x7fa0a80b1870 2023-09-30T00:48:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.178+0000 7fa0aef9a700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:00.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.178+0000 7fa0aef9a700 10 monclient(hunting): authenticate will time out at 5341.479541s 2023-09-30T00:48:00.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.184+0000 7f1ba67fc700 1 -- 172.21.15.40:0/4239756726 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1bac004be0 con 0x7f1ba00052c0 2023-09-30T00:48:00.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.184+0000 7f1ba67fc700 1 --2- 172.21.15.40:0/4239756726 >> v2:172.21.15.135:6800/116426 conn(0x7f1b98068960 0x7f1b9806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.185+0000 7f1ba67fc700 1 -- 172.21.15.40:0/4239756726 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1bac083b80 con 0x7f1ba00052c0 2023-09-30T00:48:00.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.185+0000 7f1ba67fc700 10 client.4341.objecter ms_dispatch 0x7f1bb8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:00.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.185+0000 7f1ba67fc700 3 client.4341.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:00.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.185+0000 7f1ba67fc700 3 client.4341.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:00.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.185+0000 7f1ba67fc700 20 client.4341.objecter dump_active .. 0 homeless 2023-09-30T00:48:00.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.185+0000 7f1bb5d9b700 1 --2- 172.21.15.40:0/4239756726 >> v2:172.21.15.135:6800/116426 conn(0x7f1b98068960 0x7f1b9806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:00.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.185+0000 7f1bb5d9b700 10 monclient: get_auth_request con 0x7f1b98068960 auth_method 0 2023-09-30T00:48:00.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.185+0000 7f1ba67fc700 10 client.4341.objecter ms_handle_connect 0x7f1b98068960 2023-09-30T00:48:00.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.185+0000 7f1bb5d9b700 1 --2- 172.21.15.40:0/4239756726 >> v2:172.21.15.135:6800/116426 conn(0x7f1b98068960 0x7f1b9806ae10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8006fd0 tx=0x7f1ba8006d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:00.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.186+0000 7fa0a7fff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa0a80b1870 0x7fa0a8145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:00.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.186+0000 7fec1effd700 1 -- 172.21.15.40:0/1446213468 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fec30004c20 con 0x7fec3815a560 2023-09-30T00:48:00.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.187+0000 7fec1effd700 1 --2- 172.21.15.40:0/1446213468 >> v2:172.21.15.135:6800/116426 conn(0x7fec24068960 0x7fec2406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.187+0000 7fec1effd700 1 -- 172.21.15.40:0/1446213468 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fec30083670 con 0x7fec3815a560 2023-09-30T00:48:00.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.187+0000 7fec36ffd700 1 --2- 172.21.15.40:0/1446213468 >> v2:172.21.15.135:6800/116426 conn(0x7fec24068960 0x7fec2406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:00.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.187+0000 7fec1effd700 10 client.4366.objecter ms_dispatch 0x7fec38001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:00.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.187+0000 7fec1effd700 3 client.4366.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:00.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.187+0000 7fec1effd700 3 client.4366.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:00.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.187+0000 7fec36ffd700 10 monclient: get_auth_request con 0x7fec24068960 auth_method 0 2023-09-30T00:48:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.187+0000 7fec1effd700 20 client.4366.objecter dump_active .. 0 homeless 2023-09-30T00:48:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.188+0000 7fec1effd700 10 client.4366.objecter ms_handle_connect 0x7fec24068960 2023-09-30T00:48:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.188+0000 7fec36ffd700 1 --2- 172.21.15.40:0/1446213468 >> v2:172.21.15.135:6800/116426 conn(0x7fec24068960 0x7fec2406ae10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fec2c0097c0 tx=0x7fec2c006ce0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:00.271 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.271+0000 7fe97f8cd700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:48:00.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.271+0000 7fe97f8cd700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:48:00.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.271+0000 7fe97f8cd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:00.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.271+0000 7fe97f8cd700 1 -- 172.21.15.40:0/1369260279 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe9780aa900 con 0x7fe9780ae270 2023-09-30T00:48:00.307 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.306+0000 7ff8d4ff9700 1 -- 172.21.15.40:0/2132247764 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff8d00520e0 con 0x7ff8d80af5f0 2023-09-30T00:48:00.307 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.306+0000 7ff8d4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:00.307 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.306+0000 7ff8d4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:00.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.323+0000 7f1d5ffff700 1 -- 172.21.15.40:0/3436642936 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1d6c055560 con 0x7f1d780af410 2023-09-30T00:48:00.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.323+0000 7f1d5ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:48:00.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.323+0000 7f1d5ffff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:00.324 INFO:teuthology.orchestra.run.smithi040.stdout:64424509446 2023-09-30T00:48:00.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.325+0000 7f1d7e6e7700 10 librados: watch_flush enter 2023-09-30T00:48:00.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.325+0000 7f1d7e6e7700 10 librados: watch_flush exit 2023-09-30T00:48:00.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.325+0000 7f1d7e6e7700 20 client.4301.objecter shutdown clearing up homeless session... 2023-09-30T00:48:00.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.325+0000 7f1d7e6e7700 10 client.4301.objecter successfully canceled tick 2023-09-30T00:48:00.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.325+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/3436642936 >> v2:172.21.15.135:6800/116426 conn(0x7f1d64068960 msgr2=0x7f1d6406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.325+0000 7f1d7e6e7700 1 --2- 172.21.15.40:0/3436642936 >> v2:172.21.15.135:6800/116426 conn(0x7f1d64068960 0x7f1d6406ae10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d680099b0 tx=0x7f1d68016000 comp rx=0 tx=0).stop 2023-09-30T00:48:00.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.325+0000 7f1d7e6e7700 10 monclient: shutdown 2023-09-30T00:48:00.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.325+0000 7f1d7e6e7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:00.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.325+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/3436642936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d780af410 msgr2=0x7f1d7813ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.325+0000 7f1d7e6e7700 1 --2- 172.21.15.40:0/3436642936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d780af410 0x7f1d7813ddb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c00b810 tx=0x7f1d6c009250 comp rx=0 tx=0).stop 2023-09-30T00:48:00.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.326+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/3436642936 shutdown_connections 2023-09-30T00:48:00.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.326+0000 7f1d7e6e7700 1 --2- 172.21.15.40:0/3436642936 >> v2:172.21.15.135:6800/116426 conn(0x7f1d64068960 0x7f1d6406ae10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.326+0000 7f1d7e6e7700 1 --2- 172.21.15.40:0/3436642936 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1d780b1460 0x7f1d7814d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.326+0000 7f1d7e6e7700 1 --2- 172.21.15.40:0/3436642936 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d780a5fb0 0x7f1d780a8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.326+0000 7f1d7e6e7700 1 --2- 172.21.15.40:0/3436642936 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1d780af410 0x7f1d7813ddb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.326+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/3436642936 >> 172.21.15.40:0/3436642936 conn(0x7f1d780631f0 msgr2=0x7f1d7813b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:00.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.326+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/3436642936 shutdown_connections 2023-09-30T00:48:00.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.326+0000 7f1d7e6e7700 1 -- 172.21.15.40:0/3436642936 wait complete. 2023-09-30T00:48:00.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.326+0000 7f1d7e6e7700 1 librados: shutdown 2023-09-30T00:48:00.341 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.5 2023-09-30T00:48:00.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.346+0000 7f22e85fe700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:48:00.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.346+0000 7f22e85fe700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:48:00.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.346+0000 7f22e85fe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:00.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.346+0000 7f22e85fe700 1 -- 172.21.15.40:0/4129146734 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f22c8005630 con 0x7f22e0155d90 2023-09-30T00:48:00.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.426+0000 7fec1effd700 1 -- 172.21.15.40:0/1446213468 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fec30050f90 con 0x7fec3815a560 2023-09-30T00:48:00.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.426+0000 7fec1effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:00.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.426+0000 7fec1effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:00.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.428+0000 7fa0a7fff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa0a80b1870 0x7fa0a8145b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43510/0 (socket says 172.21.15.40:43510) 2023-09-30T00:48:00.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.428+0000 7fa0a7fff700 1 -- 172.21.15.40:0/545338729 learned_addr learned my addr 172.21.15.40:0/545338729 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:00.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.428+0000 7fa0a7fff700 10 monclient(hunting): get_auth_request con 0x7fa0a80b1870 auth_method 0 2023-09-30T00:48:00.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.428+0000 7fa0a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:00.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.428+0000 7fa0a7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:00.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.428+0000 7fa0a7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:00.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.429+0000 7fa0a7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:00.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.429+0000 7fa0a7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:00.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.429+0000 7fa0a7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:00.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.431+0000 7fa0a7fff700 10 monclient(hunting): handle_auth_done global_id 4369 payload 274 2023-09-30T00:48:00.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.431+0000 7fa0a7fff700 1 -- 172.21.15.40:0/545338729 >> v2:172.21.15.40:3300/0 conn(0x7fa0a80b14a0 msgr2=0x7fa0a80afe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:00.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.431+0000 7fa0a7fff700 1 --2- 172.21.15.40:0/545338729 >> v2:172.21.15.40:3300/0 conn(0x7fa0a80b14a0 0x7fa0a80afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.431+0000 7fa0a7fff700 1 -- 172.21.15.40:0/545338729 >> v1:172.21.15.40:6789/0 conn(0x7fa0a81460b0 legacy=0x7fa0a806dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:00.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.431+0000 7fa0a7fff700 10 monclient: _finish_hunting 0 2023-09-30T00:48:00.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.431+0000 7fa0a7fff700 1 monclient: found mon.noname-c 2023-09-30T00:48:00.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.431+0000 7fa0a7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:00.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.432+0000 7fa0a7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:48:00.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.432+0000 7fa0a7fff700 1 -- 172.21.15.40:0/545338729 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a80af8f0 con 0x7fa0a80b1870 2023-09-30T00:48:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.442+0000 7f22d67fc700 1 -- 172.21.15.40:0/4129146734 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f22d001b070 con 0x7f22e0155d90 2023-09-30T00:48:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.442+0000 7f22d67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:48:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.442+0000 7f22d67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:00.443 INFO:teuthology.orchestra.run.smithi040.stdout:81604378629 2023-09-30T00:48:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.442+0000 7fa0a7fff700 1 --2- 172.21.15.40:0/545338729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0a80b1870 0x7fa0a8145b70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80b5ab0 tx=0x7fa094009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dbdc90f36c17a54 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 10 librados: watch_flush enter 2023-09-30T00:48:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 10 librados: watch_flush exit 2023-09-30T00:48:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 20 client.4363.objecter shutdown clearing up homeless session... 2023-09-30T00:48:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 10 client.4363.objecter successfully canceled tick 2023-09-30T00:48:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 1 -- 172.21.15.40:0/4129146734 >> v2:172.21.15.135:6800/116426 conn(0x7f22c4068960 msgr2=0x7f22c406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 1 --2- 172.21.15.40:0/4129146734 >> v2:172.21.15.135:6800/116426 conn(0x7f22c4068960 0x7f22c406ae10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f22d80092b0 tx=0x7f22d8017000 comp rx=0 tx=0).stop 2023-09-30T00:48:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 10 monclient: shutdown 2023-09-30T00:48:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 1 -- 172.21.15.40:0/4129146734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22e0155d90 msgr2=0x7f22e0152270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 1 --2- 172.21.15.40:0/4129146734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22e0155d90 0x7f22e0152270 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f22e006e0f0 tx=0x7f22d0002d40 comp rx=0 tx=0).stop 2023-09-30T00:48:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 1 -- 172.21.15.40:0/4129146734 shutdown_connections 2023-09-30T00:48:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 1 --2- 172.21.15.40:0/4129146734 >> v2:172.21.15.135:6800/116426 conn(0x7f22c4068960 0x7f22c406ae10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 1 --2- 172.21.15.40:0/4129146734 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f22e0155d90 0x7f22e0152270 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 1 --2- 172.21.15.40:0/4129146734 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f22cc003ae0 0x7f22e011c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 1 --2- 172.21.15.40:0/4129146734 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22e011ce80 0x7f22e0155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.445+0000 7f22e85fe700 1 -- 172.21.15.40:0/4129146734 >> 172.21.15.40:0/4129146734 conn(0x7f22e00631f0 msgr2=0x7f22e0148ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.446+0000 7f22e85fe700 1 -- 172.21.15.40:0/4129146734 shutdown_connections 2023-09-30T00:48:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.446+0000 7f22e85fe700 1 -- 172.21.15.40:0/4129146734 wait complete. 2023-09-30T00:48:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.446+0000 7f22e85fe700 1 librados: shutdown 2023-09-30T00:48:00.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0a77fe700 1 -- 172.21.15.40:0/545338729 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09400b820 con 0x7fa0a80b1870 2023-09-30T00:48:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0a77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0a77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0a77fe700 10 monclient: dump: 2023-09-30T00:48:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.452 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:00.453 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0a77fe700 10 monclient: _finish_auth 0 2023-09-30T00:48:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0a77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0a77fe700 1 -- 172.21.15.40:0/545338729 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09400b980 con 0x7fa0a80b1870 2023-09-30T00:48:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0a77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0a77fe700 1 -- 172.21.15.40:0/545338729 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09400bd20 con 0x7fa0a80b1870 2023-09-30T00:48:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0a77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0a77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0a77fe700 10 monclient: dump: 2023-09-30T00:48:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:00.453 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:00.454 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0aef9a700 5 monclient: authenticate success, global_id 4369 2023-09-30T00:48:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0aef9a700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.451+0000 7fa0aef9a700 4 set_mon_vals no callback set 2023-09-30T00:48:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.452+0000 7fa0aef9a700 10 monclient: shutdown 2023-09-30T00:48:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.452+0000 7fa0aef9a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.452+0000 7fa0aef9a700 1 -- 172.21.15.40:0/545338729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0a80b1870 msgr2=0x7fa0a8145b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.452+0000 7fa0aef9a700 1 --2- 172.21.15.40:0/545338729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0a80b1870 0x7fa0a8145b70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80b5ab0 tx=0x7fa094009250 comp rx=0 tx=0).stop 2023-09-30T00:48:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.452+0000 7fa0aef9a700 1 -- 172.21.15.40:0/545338729 shutdown_connections 2023-09-30T00:48:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.452+0000 7fa0aef9a700 1 --2- 172.21.15.40:0/545338729 >> v2:172.21.15.40:3300/0 conn(0x7fa0a80b14a0 0x7fa0a80afe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.452+0000 7fa0aef9a700 1 --2- 172.21.15.40:0/545338729 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0a80b1870 0x7fa0a8145b70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.452+0000 7fa0aef9a700 1 -- 172.21.15.40:0/545338729 >> 172.21.15.40:0/545338729 conn(0x7fa0a80631f0 msgr2=0x7fa0a80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.452+0000 7fa0aef9a700 1 -- 172.21.15.40:0/545338729 shutdown_connections 2023-09-30T00:48:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.452+0000 7fa0aef9a700 1 -- 172.21.15.40:0/545338729 wait complete. 2023-09-30T00:48:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.453+0000 7fa0aef9a700 10 monclient: build_initial_monmap 2023-09-30T00:48:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.453+0000 7fa0aef9a700 10 monclient: monmap: 2023-09-30T00:48:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:00.457 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.453+0000 7fa0aef9a700 1 librados: starting msgr at 2023-09-30T00:48:00.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.453+0000 7fa0aef9a700 1 librados: starting objecter 2023-09-30T00:48:00.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.454+0000 7fa0aef9a700 1 Processor -- start 2023-09-30T00:48:00.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.454+0000 7fa0aef9a700 1 -- start start 2023-09-30T00:48:00.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.454+0000 7fa0aef9a700 1 librados: setting wanted keys 2023-09-30T00:48:00.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.454+0000 7fa0aef9a700 1 librados: calling monclient init 2023-09-30T00:48:00.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.454+0000 7fa0aef9a700 10 monclient: init 2023-09-30T00:48:00.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:00.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:48:00.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0a80b14a0 0x7fa0a814aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 10 monclient(hunting): picked mon.noname-a con 0x7fa0a80b14a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:00.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a80b1870 0x7fa0a80ad930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 10 monclient(hunting): picked mon.noname-c con 0x7fa0a80b1870 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:00.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0a81460b0 0x7fa0a814d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 10 monclient(hunting): picked mon.noname-b con 0x7fa0a81460b0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:00.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a814c990 con 0x7fa0a80b14a0 2023-09-30T00:48:00.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a814cc00 con 0x7fa0a81460b0 2023-09-30T00:48:00.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:00.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a814cee0 con 0x7fa0a80b1870 2023-09-30T00:48:00.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:00.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.455+0000 7fa0aef9a700 10 monclient(hunting): authenticate will time out at 5341.756439s 2023-09-30T00:48:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.456+0000 7fa0ad537700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0a81460b0 0x7fa0a814d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.456+0000 7fa0ad537700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0a81460b0 0x7fa0a814d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43518/0 (socket says 172.21.15.40:43518) 2023-09-30T00:48:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.456+0000 7fa0ad537700 1 -- 172.21.15.40:0/662044472 learned_addr learned my addr 172.21.15.40:0/662044472 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.456+0000 7fa0ad537700 10 monclient(hunting): get_auth_request con 0x7fa0a81460b0 auth_method 0 2023-09-30T00:48:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.456+0000 7fa0ad537700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.456+0000 7fa0ad537700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.456+0000 7fa0ad537700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.456+0000 7fa0ad537700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.456+0000 7fa0ad537700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.456+0000 7fa0ad537700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.457+0000 7fa0a7fff700 1 --2- 172.21.15.40:0/662044472 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a80b1870 0x7fa0a80ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.457+0000 7fa0a7fff700 10 monclient(hunting): get_auth_request con 0x7fa0a80b1870 auth_method 0 2023-09-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.457+0000 7fa0a7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.457+0000 7fa0a7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.457+0000 7fa0a7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.458+0000 7fa0a7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.458+0000 7fa0a7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.458+0000 7fa0a7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.459+0000 7fa0ad537700 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2023-09-30T00:48:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.460+0000 7fa0ad537700 1 -- 172.21.15.40:0/662044472 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a80b1870 msgr2=0x7fa0a80ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.460+0000 7fa0ad537700 1 --2- 172.21.15.40:0/662044472 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a80b1870 0x7fa0a80ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.460+0000 7fa0ad537700 1 -- 172.21.15.40:0/662044472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0a80b14a0 msgr2=0x7fa0a814aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.460+0000 7fa0ad537700 1 --2- 172.21.15.40:0/662044472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0a80b14a0 0x7fa0a814aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.460+0000 7fa0ad537700 10 monclient: _finish_hunting 0 2023-09-30T00:48:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.460+0000 7fa0ad537700 1 monclient: found mon.noname-b 2023-09-30T00:48:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.460+0000 7fa0ad537700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.460+0000 7fa0ad537700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.460+0000 7fa0ad537700 1 -- 172.21.15.40:0/662044472 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0940094f0 con 0x7fa0a81460b0 2023-09-30T00:48:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.462+0000 7fa0ad537700 1 --2- 172.21.15.40:0/662044472 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0a81460b0 0x7fa0a814d300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000bbb0 tx=0x7fa0a0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.462+0000 7fa0a4ff9700 10 client.?.objecter ms_handle_connect 0x7fa0a81460b0 2023-09-30T00:48:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.462+0000 7fa0a4ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:00.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0a4ff9700 1 -- 172.21.15.40:0/662044472 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a00092e0 con 0x7fa0a81460b0 2023-09-30T00:48:00.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0a4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:00.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0a4ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:00.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0a4ff9700 10 monclient: dump: 2023-09-30T00:48:00.467 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:00.467 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.467 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.467 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.467 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:00.467 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.467 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:00.468 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0a4ff9700 10 monclient: _finish_auth 0 2023-09-30T00:48:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0a4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0a4ff9700 1 -- 172.21.15.40:0/662044472 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a000c6f0 con 0x7fa0a81460b0 2023-09-30T00:48:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0a4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0a4ff9700 1 -- 172.21.15.40:0/662044472 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a000ca90 con 0x7fa0a81460b0 2023-09-30T00:48:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0a4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0a4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0a4ff9700 10 monclient: dump: 2023-09-30T00:48:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:00.469 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0a6ffd700 4 set_mon_vals no callback set 2023-09-30T00:48:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0aef9a700 5 monclient: authenticate success, global_id 4372 2023-09-30T00:48:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0aef9a700 10 monclient: _renew_subs 2023-09-30T00:48:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0aef9a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0aef9a700 1 -- 172.21.15.40:0/662044472 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a8149320 con 0x7fa0a81460b0 2023-09-30T00:48:00.471 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378630 got 81604378629 for osd.1 2023-09-30T00:48:00.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0aef9a700 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:00.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0aef9a700 10 monclient: _renew_subs 2023-09-30T00:48:00.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0aef9a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:00.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0aef9a700 1 -- 172.21.15.40:0/662044472 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a80ade70 con 0x7fa0a81460b0 2023-09-30T00:48:00.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.466+0000 7fa0aef9a700 1 librados: init done 2023-09-30T00:48:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.468+0000 7fa0aef9a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.468+0000 7fa0aef9a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.468+0000 7fa0aef9a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.468+0000 7fa0aef9a700 1 -- 172.21.15.40:0/662044472 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a81494b0 con 0x7fa0a81460b0 2023-09-30T00:48:00.498 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.496+0000 7fa0a4ff9700 1 -- 172.21.15.40:0/662044472 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa0a0012070 con 0x7fa0a81460b0 2023-09-30T00:48:00.498 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.496+0000 7fa0a4ff9700 1 --2- 172.21.15.40:0/662044472 >> v2:172.21.15.135:6800/116426 conn(0x7fa08c068960 0x7fa08c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:00.498 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.496+0000 7fa0acd36700 1 --2- 172.21.15.40:0/662044472 >> v2:172.21.15.135:6800/116426 conn(0x7fa08c068960 0x7fa08c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:00.498 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.497+0000 7fa0acd36700 10 monclient: get_auth_request con 0x7fa08c068960 auth_method 0 2023-09-30T00:48:00.498 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.497+0000 7fa0acd36700 1 --2- 172.21.15.40:0/662044472 >> v2:172.21.15.135:6800/116426 conn(0x7fa08c068960 0x7fa08c06ae10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa09c006fd0 tx=0x7fa09c006db0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:00.498 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.497+0000 7fa0a4ff9700 10 client.4372.objecter ms_handle_connect 0x7fa08c068960 2023-09-30T00:48:00.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.520+0000 7f1ba67fc700 1 -- 172.21.15.40:0/4239756726 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1bac0515d0 con 0x7f1ba00052c0 2023-09-30T00:48:00.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.520+0000 7f1ba67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:00.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.520+0000 7f1ba67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:00.579 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.578+0000 7ff8df01c700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:48:00.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.578+0000 7ff8df01c700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:48:00.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.578+0000 7ff8df01c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:48:00.580 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.578+0000 7ff8df01c700 1 -- 172.21.15.40:0/2132247764 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff8d8064a00 con 0x7ff8d80af5f0 2023-09-30T00:48:00.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.586+0000 7ff8d4ff9700 1 -- 172.21.15.40:0/2132247764 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff8d0055700 con 0x7ff8d80af5f0 2023-09-30T00:48:00.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.586+0000 7ff8d4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:48:00.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.586+0000 7ff8d4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:00.587 INFO:teuthology.orchestra.run.smithi040.stdout:85899345925 2023-09-30T00:48:00.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 10 librados: watch_flush enter 2023-09-30T00:48:00.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 10 librados: watch_flush exit 2023-09-30T00:48:00.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 20 client.4307.objecter shutdown clearing up homeless session... 2023-09-30T00:48:00.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 10 client.4307.objecter successfully canceled tick 2023-09-30T00:48:00.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 1 -- 172.21.15.40:0/2132247764 >> v2:172.21.15.135:6800/116426 conn(0x7ff8bc0689e0 msgr2=0x7ff8bc06ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 1 --2- 172.21.15.40:0/2132247764 >> v2:172.21.15.135:6800/116426 conn(0x7ff8bc0689e0 0x7ff8bc06ae90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009720 tx=0x7ff8cc006c30 comp rx=0 tx=0).stop 2023-09-30T00:48:00.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 10 monclient: shutdown 2023-09-30T00:48:00.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:00.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 1 -- 172.21.15.40:0/2132247764 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8d80af5f0 msgr2=0x7ff8d80a9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 1 --2- 172.21.15.40:0/2132247764 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8d80af5f0 0x7ff8d80a9c40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff8d000bb60 tx=0x7ff8d0009810 comp rx=0 tx=0).stop 2023-09-30T00:48:00.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 1 -- 172.21.15.40:0/2132247764 shutdown_connections 2023-09-30T00:48:00.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 1 --2- 172.21.15.40:0/2132247764 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff8d80af5f0 0x7ff8d80a9c40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 1 --2- 172.21.15.40:0/2132247764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8d8068b10 0x7ff8d80a9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 1 --2- 172.21.15.40:0/2132247764 >> v2:172.21.15.135:6800/116426 conn(0x7ff8bc0689e0 0x7ff8bc06ae90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 1 --2- 172.21.15.40:0/2132247764 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff8d80adfe0 0x7ff8d80abce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 1 -- 172.21.15.40:0/2132247764 >> 172.21.15.40:0/2132247764 conn(0x7ff8d8067a00 msgr2=0x7ff8d80b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:00.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 1 -- 172.21.15.40:0/2132247764 shutdown_connections 2023-09-30T00:48:00.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.588+0000 7ff8df01c700 1 -- 172.21.15.40:0/2132247764 wait complete. 2023-09-30T00:48:00.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.589+0000 7ff8df01c700 1 librados: shutdown 2023-09-30T00:48:00.609 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345927 got 85899345925 for osd.0 2023-09-30T00:48:00.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.634+0000 7fe9757fa700 1 -- 172.21.15.40:0/1369260279 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe9780aa900 con 0x7fe9780ae270 2023-09-30T00:48:00.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.634+0000 7fe9757fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:48:00.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.634+0000 7fe9757fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:00.635 INFO:teuthology.orchestra.run.smithi040.stdout:85899345927 2023-09-30T00:48:00.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.636+0000 7fe97f8cd700 10 librados: watch_flush enter 2023-09-30T00:48:00.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.636+0000 7fe97f8cd700 10 librados: watch_flush exit 2023-09-30T00:48:00.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.636+0000 7fe97f8cd700 20 client.4332.objecter shutdown clearing up homeless session... 2023-09-30T00:48:00.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.636+0000 7fe97f8cd700 10 client.4332.objecter successfully canceled tick 2023-09-30T00:48:00.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.636+0000 7fe97f8cd700 1 -- 172.21.15.40:0/1369260279 >> v2:172.21.15.135:6800/116426 conn(0x7fe964068890 msgr2=0x7fe96406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.636+0000 7fe97f8cd700 1 --2- 172.21.15.40:0/1369260279 >> v2:172.21.15.135:6800/116426 conn(0x7fe964068890 0x7fe96406ad40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe968006fd0 tx=0x7fe968006d30 comp rx=0 tx=0).stop 2023-09-30T00:48:00.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.636+0000 7fe97f8cd700 10 monclient: shutdown 2023-09-30T00:48:00.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.636+0000 7fe97f8cd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:00.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.636+0000 7fe97f8cd700 1 -- 172.21.15.40:0/1369260279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9780ae270 msgr2=0x7fe9780ae6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.636+0000 7fe97f8cd700 1 --2- 172.21.15.40:0/1369260279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9780ae270 0x7fe9780ae6e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe9780b0d20 tx=0x7fe970007e10 comp rx=0 tx=0).stop 2023-09-30T00:48:00.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.637+0000 7fe97f8cd700 1 -- 172.21.15.40:0/1369260279 shutdown_connections 2023-09-30T00:48:00.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.637+0000 7fe97f8cd700 1 --2- 172.21.15.40:0/1369260279 >> v2:172.21.15.135:6800/116426 conn(0x7fe964068890 0x7fe96406ad40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.637+0000 7fe97f8cd700 1 --2- 172.21.15.40:0/1369260279 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe960004810 0x7fe97813e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.637+0000 7fe97f8cd700 1 --2- 172.21.15.40:0/1369260279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9780ae270 0x7fe9780ae6e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.637+0000 7fe97f8cd700 1 --2- 172.21.15.40:0/1369260279 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9780a9910 0x7fe9780a9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.637+0000 7fe97f8cd700 1 -- 172.21.15.40:0/1369260279 >> 172.21.15.40:0/1369260279 conn(0x7fe9780631f0 msgr2=0x7fe9780afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:00.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.637+0000 7fe97f8cd700 1 -- 172.21.15.40:0/1369260279 shutdown_connections 2023-09-30T00:48:00.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.637+0000 7fe97f8cd700 1 -- 172.21.15.40:0/1369260279 wait complete. 2023-09-30T00:48:00.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.637+0000 7fe97f8cd700 1 librados: shutdown 2023-09-30T00:48:00.652 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345927 got 85899345927 for osd.3 2023-09-30T00:48:00.652 DEBUG:teuthology.parallel:result is None 2023-09-30T00:48:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.664+0000 7fa0a4ff9700 1 -- 172.21.15.40:0/662044472 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa0a0055e30 con 0x7fa0a81460b0 2023-09-30T00:48:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.664+0000 7fa0a4ff9700 10 client.4372.objecter ms_dispatch 0x7fa0a8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.664+0000 7fa0a4ff9700 3 client.4372.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.664+0000 7fa0a4ff9700 3 client.4372.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.664+0000 7fa0a4ff9700 20 client.4372.objecter dump_active .. 0 homeless 2023-09-30T00:48:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.667+0000 7fec3d371700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:48:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.667+0000 7fec3d371700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:48:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.667+0000 7fec3d371700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.667+0000 7fec3d371700 1 -- 172.21.15.40:0/1446213468 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fec38064a00 con 0x7fec3815a560 2023-09-30T00:48:00.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.744+0000 7f1bbc9f8700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T00:48:00.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.744+0000 7f1bbc9f8700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T00:48:00.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.744+0000 7f1bbc9f8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:00.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.744+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/4239756726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f1b9c005630 con 0x7f1ba00052c0 2023-09-30T00:48:00.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.750+0000 7f1ba67fc700 1 -- 172.21.15.40:0/4239756726 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1bac022020 con 0x7f1ba00052c0 2023-09-30T00:48:00.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.750+0000 7f1ba67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T00:48:00.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.750+0000 7f1ba67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:00.751 INFO:teuthology.orchestra.run.smithi040.stdout:60129542151 2023-09-30T00:48:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.752+0000 7f1bbc9f8700 10 librados: watch_flush enter 2023-09-30T00:48:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.752+0000 7f1bbc9f8700 10 librados: watch_flush exit 2023-09-30T00:48:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 20 client.4341.objecter shutdown clearing up homeless session... 2023-09-30T00:48:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 10 client.4341.objecter successfully canceled tick 2023-09-30T00:48:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/4239756726 >> v2:172.21.15.135:6800/116426 conn(0x7f1b98068960 msgr2=0x7f1b9806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 1 --2- 172.21.15.40:0/4239756726 >> v2:172.21.15.135:6800/116426 conn(0x7f1b98068960 0x7f1b9806ae10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8006fd0 tx=0x7f1ba8006d30 comp rx=0 tx=0).stop 2023-09-30T00:48:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 10 monclient: shutdown 2023-09-30T00:48:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/4239756726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba00052c0 msgr2=0x7f1bb813dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 1 --2- 172.21.15.40:0/4239756726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba00052c0 0x7f1bb813dfa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1bac00c930 tx=0x7f1bac001b80 comp rx=0 tx=0).stop 2023-09-30T00:48:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/4239756726 shutdown_connections 2023-09-30T00:48:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 1 --2- 172.21.15.40:0/4239756726 >> v2:172.21.15.135:6800/116426 conn(0x7f1b98068960 0x7f1b9806ae10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 1 --2- 172.21.15.40:0/4239756726 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1bb8151fe0 0x7f1bb8152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 1 --2- 172.21.15.40:0/4239756726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1ba00052c0 0x7f1bb813dfa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 1 --2- 172.21.15.40:0/4239756726 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1bb806c860 0x7f1bb806d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/4239756726 >> 172.21.15.40:0/4239756726 conn(0x7f1bb80631f0 msgr2=0x7f1bb8144aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/4239756726 shutdown_connections 2023-09-30T00:48:00.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 1 -- 172.21.15.40:0/4239756726 wait complete. 2023-09-30T00:48:00.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.753+0000 7f1bbc9f8700 1 librados: shutdown 2023-09-30T00:48:00.767 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.6 2023-09-30T00:48:00.768 DEBUG:teuthology.parallel:result is None 2023-09-30T00:48:00.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.807+0000 7fa0a4ff9700 1 -- 172.21.15.40:0/662044472 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa0a0051a70 con 0x7fa0a81460b0 2023-09-30T00:48:00.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.807+0000 7fa0a4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:00.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.807+0000 7fa0a4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:00.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.824+0000 7fec1effd700 1 -- 172.21.15.40:0/1446213468 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fec3001a090 con 0x7fec3815a560 2023-09-30T00:48:00.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.824+0000 7fec1effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:48:00.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.824+0000 7fec1effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:00.826 INFO:teuthology.orchestra.run.smithi040.stdout:64424509446 2023-09-30T00:48:00.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 10 librados: watch_flush enter 2023-09-30T00:48:00.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 10 librados: watch_flush exit 2023-09-30T00:48:00.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 20 client.4366.objecter shutdown clearing up homeless session... 2023-09-30T00:48:00.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 10 client.4366.objecter successfully canceled tick 2023-09-30T00:48:00.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 1 -- 172.21.15.40:0/1446213468 >> v2:172.21.15.135:6800/116426 conn(0x7fec24068960 msgr2=0x7fec2406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 1 --2- 172.21.15.40:0/1446213468 >> v2:172.21.15.135:6800/116426 conn(0x7fec24068960 0x7fec2406ae10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fec2c0097c0 tx=0x7fec2c006ce0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 10 monclient: shutdown 2023-09-30T00:48:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 1 -- 172.21.15.40:0/1446213468 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec3815a560 msgr2=0x7fec3815a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 1 --2- 172.21.15.40:0/1446213468 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec3815a560 0x7fec3815a9e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fec380af150 tx=0x7fec30003030 comp rx=0 tx=0).stop 2023-09-30T00:48:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 1 -- 172.21.15.40:0/1446213468 shutdown_connections 2023-09-30T00:48:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 1 --2- 172.21.15.40:0/1446213468 >> v2:172.21.15.135:6800/116426 conn(0x7fec24068960 0x7fec2406ae10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 1 --2- 172.21.15.40:0/1446213468 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec3815a560 0x7fec3815a9e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 1 --2- 172.21.15.40:0/1446213468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec20004810 0x7fec380b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 1 --2- 172.21.15.40:0/1446213468 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec380a54c0 0x7fec380ac650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 1 -- 172.21.15.40:0/1446213468 >> 172.21.15.40:0/1446213468 conn(0x7fec3806af40 msgr2=0x7fec3806db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:00.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 1 -- 172.21.15.40:0/1446213468 shutdown_connections 2023-09-30T00:48:00.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 1 -- 172.21.15.40:0/1446213468 wait complete. 2023-09-30T00:48:00.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:00.827+0000 7fec3d371700 1 librados: shutdown 2023-09-30T00:48:00.842 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.4 2023-09-30T00:48:00.843 DEBUG:teuthology.parallel:result is None 2023-09-30T00:48:01.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.041+0000 7fa0aef9a700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:48:01.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.041+0000 7fa0aef9a700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:48:01.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.041+0000 7fa0aef9a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:01.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.041+0000 7fa0aef9a700 1 -- 172.21.15.40:0/662044472 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa0a814d0a0 con 0x7fa0a81460b0 2023-09-30T00:48:01.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.046+0000 7fa0a4ff9700 1 -- 172.21.15.40:0/662044472 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa0a0055090 con 0x7fa0a81460b0 2023-09-30T00:48:01.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.046+0000 7fa0a4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:48:01.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.046+0000 7fa0a4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:01.046 INFO:teuthology.orchestra.run.smithi040.stdout:85899345926 2023-09-30T00:48:01.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.047+0000 7fa0aef9a700 10 librados: watch_flush enter 2023-09-30T00:48:01.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.047+0000 7fa0aef9a700 10 librados: watch_flush exit 2023-09-30T00:48:01.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 20 client.4372.objecter shutdown clearing up homeless session... 2023-09-30T00:48:01.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 10 client.4372.objecter successfully canceled tick 2023-09-30T00:48:01.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 1 -- 172.21.15.40:0/662044472 >> v2:172.21.15.135:6800/116426 conn(0x7fa08c068960 msgr2=0x7fa08c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 1 --2- 172.21.15.40:0/662044472 >> v2:172.21.15.135:6800/116426 conn(0x7fa08c068960 0x7fa08c06ae10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa09c006fd0 tx=0x7fa09c006db0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 10 monclient: shutdown 2023-09-30T00:48:01.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:01.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 1 -- 172.21.15.40:0/662044472 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0a81460b0 msgr2=0x7fa0a814d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 1 --2- 172.21.15.40:0/662044472 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0a81460b0 0x7fa0a814d300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000bbb0 tx=0x7fa0a0009840 comp rx=0 tx=0).stop 2023-09-30T00:48:01.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 1 -- 172.21.15.40:0/662044472 shutdown_connections 2023-09-30T00:48:01.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 1 --2- 172.21.15.40:0/662044472 >> v2:172.21.15.135:6800/116426 conn(0x7fa08c068960 0x7fa08c06ae10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 1 --2- 172.21.15.40:0/662044472 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa0a81460b0 0x7fa0a814d300 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 1 --2- 172.21.15.40:0/662044472 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0a80b14a0 0x7fa0a814aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 1 --2- 172.21.15.40:0/662044472 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa0a80b1870 0x7fa0a80ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 1 -- 172.21.15.40:0/662044472 >> 172.21.15.40:0/662044472 conn(0x7fa0a80631f0 msgr2=0x7fa0a80635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:01.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 1 -- 172.21.15.40:0/662044472 shutdown_connections 2023-09-30T00:48:01.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 1 -- 172.21.15.40:0/662044472 wait complete. 2023-09-30T00:48:01.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.048+0000 7fa0aef9a700 1 librados: shutdown 2023-09-30T00:48:01.063 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345926 got 85899345926 for osd.2 2023-09-30T00:48:01.064 DEBUG:teuthology.parallel:result is None 2023-09-30T00:48:01.172 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T00:48:01.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.310+0000 7f508a0b2700 10 monclient: get_monmap_and_config 2023-09-30T00:48:01.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.310+0000 7f508a0b2700 10 monclient: build_initial_monmap 2023-09-30T00:48:01.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.310+0000 7f508a0b2700 10 monclient: monmap: 2023-09-30T00:48:01.311 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:01.311 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.311 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:01.311284+0000 2023-09-30T00:48:01.311 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:01.311284+0000 2023-09-30T00:48:01.311 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:01.311 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.311 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:01.312 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:01.312 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:01.312 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:01.312 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:01.312 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 1 Processor -- start 2023-09-30T00:48:01.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 1 -- start start 2023-09-30T00:48:01.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 10 monclient: init 2023-09-30T00:48:01.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:01.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-09-30T00:48:01.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50840b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:01.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f50840b1830 0x7f50840a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 10 monclient(hunting): picked mon.noname-a con 0x7f50840b1830 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:01.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f508413ac20 addr v1:172.21.15.40:6789/0 2023-09-30T00:48:01.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50840a5b20 con 0x7f508413ac20 2023-09-30T00:48:01.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50840a71f0 con 0x7f50840b1460 2023-09-30T00:48:01.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5084141e40 con 0x7f50840b1830 2023-09-30T00:48:01.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:01.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.311+0000 7f508a0b2700 10 monclient(hunting): authenticate will time out at 5342.612854s 2023-09-30T00:48:01.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.312+0000 7f5082ffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f50840b1830 0x7f50840a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:01.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.312+0000 7f50837fe700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f50840b1460 0x7f5084141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:54112/0 (socket says 172.21.15.40:54112) 2023-09-30T00:48:01.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.312+0000 7f50837fe700 1 -- 172.21.15.40:0/594897839 learned_addr learned my addr 172.21.15.40:0/594897839 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:48:01.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.313+0000 7f5082ffd700 10 monclient(hunting): get_auth_request con 0x7f50840b1830 auth_method 0 2023-09-30T00:48:01.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.313+0000 7f5082ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:01.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.313+0000 7f5082ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:01.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.313+0000 7f5082ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:01.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.313+0000 7f5082ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:01.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.313+0000 7f5082ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:01.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.313+0000 7f5082ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:01.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.316+0000 7f5082ffd700 10 monclient(hunting): handle_auth_done global_id 4344 payload 274 2023-09-30T00:48:01.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.316+0000 7f5082ffd700 1 -- 172.21.15.40:0/594897839 >> v1:172.21.15.135:6789/0 conn(0x7f50840b1460 legacy=0x7f5084141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.316+0000 7f5082ffd700 1 -- 172.21.15.40:0/594897839 >> v1:172.21.15.40:6789/0 conn(0x7f508413ac20 legacy=0x7f508413b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:01.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.316+0000 7f5082ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:48:01.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.316+0000 7f5082ffd700 1 monclient: found mon.noname-a 2023-09-30T00:48:01.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.316+0000 7f5082ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:01.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.316+0000 7f5082ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:01.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.316+0000 7f5082ffd700 1 -- 172.21.15.40:0/594897839 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50840a74d0 con 0x7f50840b1830 2023-09-30T00:48:01.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.320+0000 7f5082ffd700 1 --2- 172.21.15.40:0/594897839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50840b1830 0x7f50840a5fb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f506c009910 tx=0x7f506c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5daf286cb652dd34 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:01.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f50827fc700 1 -- 172.21.15.40:0/594897839 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c00b7e0 con 0x7f50840b1830 2023-09-30T00:48:01.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f50827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f50827fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:01.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f50827fc700 10 monclient: dump: 2023-09-30T00:48:01.327 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.328 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f50827fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f50827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:01.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f50827fc700 1 -- 172.21.15.40:0/594897839 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506c00b940 con 0x7f50840b1830 2023-09-30T00:48:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f50827fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f50827fc700 1 -- 172.21.15.40:0/594897839 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c00bce0 con 0x7f50840b1830 2023-09-30T00:48:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f508a0b2700 5 monclient: authenticate success, global_id 4344 2023-09-30T00:48:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f508a0b2700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f50827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f508a0b2700 4 set_mon_vals no callback set 2023-09-30T00:48:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f50827fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.326+0000 7f50827fc700 10 monclient: dump: 2023-09-30T00:48:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.329 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.330 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.330 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.327+0000 7f508a0b2700 10 monclient: shutdown 2023-09-30T00:48:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.327+0000 7f508a0b2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.327+0000 7f508a0b2700 1 -- 172.21.15.40:0/594897839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50840b1830 msgr2=0x7f50840a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.327+0000 7f508a0b2700 1 --2- 172.21.15.40:0/594897839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50840b1830 0x7f50840a5fb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f506c009910 tx=0x7f506c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.327+0000 7f508a0b2700 1 -- 172.21.15.40:0/594897839 shutdown_connections 2023-09-30T00:48:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.327+0000 7f508a0b2700 1 --2- 172.21.15.40:0/594897839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50840b1830 0x7f50840a5fb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.327+0000 7f508a0b2700 1 -- 172.21.15.40:0/594897839 >> 172.21.15.40:0/594897839 conn(0x7f50840631f0 msgr2=0x7f50840635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.327+0000 7f508a0b2700 1 -- 172.21.15.40:0/594897839 shutdown_connections 2023-09-30T00:48:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.327+0000 7f508a0b2700 1 -- 172.21.15.40:0/594897839 wait complete. 2023-09-30T00:48:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.328+0000 7f508a0b2700 10 monclient: build_initial_monmap 2023-09-30T00:48:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.328+0000 7f508a0b2700 10 monclient: monmap: 2023-09-30T00:48:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:01.334 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:01.335 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.328+0000 7f508a0b2700 1 librados: starting msgr at 2023-09-30T00:48:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.328+0000 7f508a0b2700 1 librados: starting objecter 2023-09-30T00:48:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.329+0000 7f508a0b2700 1 Processor -- start 2023-09-30T00:48:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.329+0000 7f508a0b2700 1 -- start start 2023-09-30T00:48:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.329+0000 7f508a0b2700 1 librados: setting wanted keys 2023-09-30T00:48:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.329+0000 7f508a0b2700 1 librados: calling monclient init 2023-09-30T00:48:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.329+0000 7f508a0b2700 10 monclient: init 2023-09-30T00:48:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.329+0000 7f508a0b2700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.329+0000 7f508a0b2700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:48:01.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.329+0000 7f508a0b2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50840b1460 0x7f50840a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.329+0000 7f508a0b2700 10 monclient(hunting): picked mon.noname-b con 0x7f50840b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:01.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.329+0000 7f508a0b2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50840b1830 0x7f50840a86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.329+0000 7f508a0b2700 10 monclient(hunting): picked mon.noname-a con 0x7f50840b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:01.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.330+0000 7f508a0b2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f508413ac20 0x7f508413d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.330+0000 7f508a0b2700 10 monclient(hunting): picked mon.noname-c con 0x7f508413ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:01.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.330+0000 7f508a0b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.330+0000 7f508a0b2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50840a93c0 con 0x7f50840b1830 2023-09-30T00:48:01.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.330+0000 7f508a0b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.330+0000 7f508a0b2700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508413f050 con 0x7f50840b1460 2023-09-30T00:48:01.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.330+0000 7f508a0b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.330+0000 7f508a0b2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50840ae6e0 con 0x7f508413ac20 2023-09-30T00:48:01.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.330+0000 7f508a0b2700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:01.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.330+0000 7f508a0b2700 10 monclient(hunting): authenticate will time out at 5342.631044s 2023-09-30T00:48:01.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.330+0000 7f5083fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f508413ac20 0x7f508413d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:01.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.330+0000 7f50837fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50840b1460 0x7f50840a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:01.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.330+0000 7f5082ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50840b1830 0x7f50840a86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:01.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f50837fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50840b1460 0x7f50840a8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43534/0 (socket says 172.21.15.40:43534) 2023-09-30T00:48:01.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f50837fe700 1 -- 172.21.15.40:0/645373248 learned_addr learned my addr 172.21.15.40:0/645373248 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:01.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f50837fe700 10 monclient(hunting): get_auth_request con 0x7f50840b1460 auth_method 0 2023-09-30T00:48:01.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f50837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:01.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f50837fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:01.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f50837fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:01.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f5082ffd700 10 monclient(hunting): get_auth_request con 0x7f50840b1830 auth_method 0 2023-09-30T00:48:01.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f5082ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:01.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f5082ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:01.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f5082ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:01.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f5083fff700 10 monclient(hunting): get_auth_request con 0x7f508413ac20 auth_method 0 2023-09-30T00:48:01.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f5083fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:01.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f5083fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:01.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f5083fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:01.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f50837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:01.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f50837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:01.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f50837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:01.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f5082ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:01.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f5082ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:01.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.331+0000 7f5082ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:01.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.332+0000 7f5083fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:01.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.332+0000 7f5083fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:01.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.332+0000 7f5083fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.335+0000 7f50837fe700 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2023-09-30T00:48:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.335+0000 7f50837fe700 1 -- 172.21.15.40:0/645373248 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f508413ac20 msgr2=0x7f508413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.335+0000 7f50837fe700 1 --2- 172.21.15.40:0/645373248 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f508413ac20 0x7f508413d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.335+0000 7f50837fe700 1 -- 172.21.15.40:0/645373248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50840b1830 msgr2=0x7f50840a86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.335+0000 7f50837fe700 1 --2- 172.21.15.40:0/645373248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50840b1830 0x7f50840a86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.335+0000 7f50837fe700 10 monclient: _finish_hunting 0 2023-09-30T00:48:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.335+0000 7f50837fe700 1 monclient: found mon.noname-b 2023-09-30T00:48:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.335+0000 7f50837fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:01.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.335+0000 7f50837fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:01.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.335+0000 7f50837fe700 1 -- 172.21.15.40:0/645373248 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c009420 con 0x7f50840b1460 2023-09-30T00:48:01.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.337+0000 7f50837fe700 1 --2- 172.21.15.40:0/645373248 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50840b1460 0x7f50840a8d30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f507400c7c0 tx=0x7f5074007f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:01.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.337+0000 7f507b7fe700 10 client.?.objecter ms_handle_connect 0x7f50840b1460 2023-09-30T00:48:01.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.337+0000 7f507b7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f507b7fe700 1 -- 172.21.15.40:0/645373248 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074005b30 con 0x7f50840b1460 2023-09-30T00:48:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f507b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f507b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f507b7fe700 10 monclient: dump: 2023-09-30T00:48:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.344 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.344 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f507b7fe700 10 monclient: _finish_auth 0 2023-09-30T00:48:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f507b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f507b7fe700 1 -- 172.21.15.40:0/645373248 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5074005c90 con 0x7f50840b1460 2023-09-30T00:48:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f507b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f507b7fe700 1 -- 172.21.15.40:0/645373248 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074004a80 con 0x7f50840b1460 2023-09-30T00:48:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f507b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f507b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f507b7fe700 10 monclient: dump: 2023-09-30T00:48:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.345 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.346 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f508a0b2700 5 monclient: authenticate success, global_id 4378 2023-09-30T00:48:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f508a0b2700 10 monclient: _renew_subs 2023-09-30T00:48:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f508a0b2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f508a0b2700 1 -- 172.21.15.40:0/645373248 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50840a9850 con 0x7f50840b1460 2023-09-30T00:48:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f5081ffb700 4 set_mon_vals no callback set 2023-09-30T00:48:01.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f508a0b2700 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:01.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f508a0b2700 10 monclient: _renew_subs 2023-09-30T00:48:01.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f508a0b2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:01.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f508a0b2700 1 -- 172.21.15.40:0/645373248 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508413d9f0 con 0x7f50840b1460 2023-09-30T00:48:01.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.342+0000 7f508a0b2700 1 librados: init done 2023-09-30T00:48:01.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.343+0000 7f508a0b2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:01.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.343+0000 7f508a0b2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:01.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.343+0000 7f508a0b2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:01.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.343+0000 7f508a0b2700 1 -- 172.21.15.40:0/645373248 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50840a96a0 con 0x7f50840b1460 2023-09-30T00:48:01.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T00:48:01.472 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:48:01.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.479+0000 7fee4371a700 10 monclient: get_monmap_and_config 2023-09-30T00:48:01.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.479+0000 7fee4371a700 10 monclient: build_initial_monmap 2023-09-30T00:48:01.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.480+0000 7fee4371a700 10 monclient: monmap: 2023-09-30T00:48:01.481 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:01.481 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.481 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:01.481068+0000 2023-09-30T00:48:01.481 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:01.481068+0000 2023-09-30T00:48:01.481 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:01.481 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.481 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:01.481 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:01.481 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:01.481 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:01.481 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:01.482 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.480+0000 7fee4371a700 1 Processor -- start 2023-09-30T00:48:01.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.480+0000 7fee4371a700 1 -- start start 2023-09-30T00:48:01.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.480+0000 7fee4371a700 10 monclient: init 2023-09-30T00:48:01.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.481+0000 7fee4371a700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:01.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.481+0000 7fee4371a700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-09-30T00:48:01.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.481+0000 7fee4371a700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fee3c0b1460 0x7fee3c0b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.481+0000 7fee4371a700 10 monclient(hunting): picked mon.noname-c con 0x7fee3c0b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:01.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.481+0000 7fee4371a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fee3c0a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:01.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.481+0000 7fee4371a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fee3c13ac20 0x7fee3c13b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.481+0000 7fee4371a700 10 monclient(hunting): picked mon.noname-a con 0x7fee3c13ac20 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:01.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.481+0000 7fee4371a700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3c0a5b20 con 0x7fee3c0a5fb0 2023-09-30T00:48:01.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.481+0000 7fee4371a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.481+0000 7fee4371a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee3c0a71f0 con 0x7fee3c13ac20 2023-09-30T00:48:01.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.481+0000 7fee4371a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.481+0000 7fee4371a700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee3c0a74d0 con 0x7fee3c0b1460 2023-09-30T00:48:01.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.481+0000 7fee4371a700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:01.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.481+0000 7fee4371a700 10 monclient(hunting): authenticate will time out at 5342.782621s 2023-09-30T00:48:01.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.482+0000 7fee41cb7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fee3c13ac20 0x7fee3c13b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:01.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.482+0000 7fee41cb7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fee3c13ac20 0x7fee3c13b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52330/0 (socket says 172.21.15.40:52330) 2023-09-30T00:48:01.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.482+0000 7fee41cb7700 1 -- 172.21.15.40:0/345850035 learned_addr learned my addr 172.21.15.40:0/345850035 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:01.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.482+0000 7fee41cb7700 10 monclient(hunting): get_auth_request con 0x7fee3c13ac20 auth_method 0 2023-09-30T00:48:01.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.482+0000 7fee41cb7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:01.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.482+0000 7fee41cb7700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:01.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.482+0000 7fee41cb7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:01.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.483+0000 7fee41cb7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:01.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.483+0000 7fee41cb7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:01.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.483+0000 7fee41cb7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:01.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.485+0000 7fee41cb7700 10 monclient(hunting): handle_auth_done global_id 4350 payload 274 2023-09-30T00:48:01.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.485+0000 7fee41cb7700 1 -- 172.21.15.40:0/345850035 >> v2:172.21.15.135:3300/0 conn(0x7fee3c0b1460 msgr2=0x7fee3c0b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:01.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.485+0000 7fee41cb7700 1 --2- 172.21.15.40:0/345850035 >> v2:172.21.15.135:3300/0 conn(0x7fee3c0b1460 0x7fee3c0b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.485+0000 7fee41cb7700 1 -- 172.21.15.40:0/345850035 >> v1:172.21.15.135:6789/0 conn(0x7fee3c0a5fb0 legacy=0x7fee3c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:01.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.485+0000 7fee41cb7700 10 monclient: _finish_hunting 0 2023-09-30T00:48:01.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.485+0000 7fee41cb7700 1 monclient: found mon.noname-a 2023-09-30T00:48:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.485+0000 7fee41cb7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.485+0000 7fee41cb7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.485+0000 7fee41cb7700 1 -- 172.21.15.40:0/345850035 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3c0a63c0 con 0x7fee3c13ac20 2023-09-30T00:48:01.487 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.487+0000 7fee41cb7700 1 --2- 172.21.15.40:0/345850035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee3c13ac20 0x7fee3c13b080 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fee34009f20 tx=0x7fee34009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce62685e03f3b8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:01.492 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee33fff700 1 -- 172.21.15.40:0/345850035 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3400e070 con 0x7fee3c13ac20 2023-09-30T00:48:01.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee33fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee33fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:01.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee33fff700 10 monclient: dump: 2023-09-30T00:48:01.493 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.493 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.493 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.493 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.493 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.493 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.493 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.494 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.494 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.494 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee33fff700 10 monclient: _finish_auth 0 2023-09-30T00:48:01.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee33fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:01.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee33fff700 1 -- 172.21.15.40:0/345850035 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee3400ba00 con 0x7fee3c13ac20 2023-09-30T00:48:01.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee33fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:01.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee33fff700 1 -- 172.21.15.40:0/345850035 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee340092e0 con 0x7fee3c13ac20 2023-09-30T00:48:01.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee4371a700 5 monclient: authenticate success, global_id 4350 2023-09-30T00:48:01.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee4371a700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:01.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee4371a700 4 set_mon_vals no callback set 2023-09-30T00:48:01.495 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee33fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.495 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee33fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:01.495 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.492+0000 7fee33fff700 10 monclient: dump: 2023-09-30T00:48:01.495 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.495 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.495 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.495 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.495 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.495 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.495 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.495 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.495 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.496 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.498 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.493+0000 7fee4371a700 10 monclient: shutdown 2023-09-30T00:48:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.493+0000 7fee4371a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.493+0000 7fee4371a700 1 -- 172.21.15.40:0/345850035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee3c13ac20 msgr2=0x7fee3c13b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.493+0000 7fee4371a700 1 --2- 172.21.15.40:0/345850035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee3c13ac20 0x7fee3c13b080 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fee34009f20 tx=0x7fee34009840 comp rx=0 tx=0).stop 2023-09-30T00:48:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.493+0000 7fee4371a700 1 -- 172.21.15.40:0/345850035 shutdown_connections 2023-09-30T00:48:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.493+0000 7fee4371a700 1 --2- 172.21.15.40:0/345850035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee3c13ac20 0x7fee3c13b080 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.493+0000 7fee4371a700 1 --2- 172.21.15.40:0/345850035 >> v2:172.21.15.135:3300/0 conn(0x7fee3c0b1460 0x7fee3c0b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.493+0000 7fee4371a700 1 -- 172.21.15.40:0/345850035 >> 172.21.15.40:0/345850035 conn(0x7fee3c0631f0 msgr2=0x7fee3c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:01.499 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.493+0000 7fee4371a700 1 -- 172.21.15.40:0/345850035 shutdown_connections 2023-09-30T00:48:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.493+0000 7fee4371a700 1 -- 172.21.15.40:0/345850035 wait complete. 2023-09-30T00:48:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.494+0000 7fee4371a700 10 monclient: build_initial_monmap 2023-09-30T00:48:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.494+0000 7fee4371a700 10 monclient: monmap: 2023-09-30T00:48:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:01.500 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:01.501 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.495+0000 7fee4371a700 1 librados: starting msgr at 2023-09-30T00:48:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.495+0000 7fee4371a700 1 librados: starting objecter 2023-09-30T00:48:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.495+0000 7fee4371a700 1 Processor -- start 2023-09-30T00:48:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.495+0000 7fee4371a700 1 -- start start 2023-09-30T00:48:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.495+0000 7fee4371a700 1 librados: setting wanted keys 2023-09-30T00:48:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.495+0000 7fee4371a700 1 librados: calling monclient init 2023-09-30T00:48:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.495+0000 7fee4371a700 10 monclient: init 2023-09-30T00:48:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:48:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee3c0a5fb0 0x7fee3c0aa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 10 monclient(hunting): picked mon.noname-c con 0x7fee3c0a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:01.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee3c0b1460 0x7fee3c13e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 10 monclient(hunting): picked mon.noname-b con 0x7fee3c0b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee3c13ac20 0x7fee3c151120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 10 monclient(hunting): picked mon.noname-a con 0x7fee3c13ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c0a9f50 con 0x7fee3c13ac20 2023-09-30T00:48:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c0ab8c0 con 0x7fee3c0b1460 2023-09-30T00:48:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c13f2e0 con 0x7fee3c0a5fb0 2023-09-30T00:48:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:01.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.496+0000 7fee4371a700 10 monclient(hunting): authenticate will time out at 5342.797604s 2023-09-30T00:48:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.497+0000 7fee41cb7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee3c13ac20 0x7fee3c151120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.497+0000 7fee414b6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee3c0a5fb0 0x7fee3c0aa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.497+0000 7fee41cb7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee3c13ac20 0x7fee3c151120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52338/0 (socket says 172.21.15.40:52338) 2023-09-30T00:48:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.497+0000 7fee41cb7700 1 -- 172.21.15.40:0/3326160130 learned_addr learned my addr 172.21.15.40:0/3326160130 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.497+0000 7fee41cb7700 10 monclient(hunting): get_auth_request con 0x7fee3c13ac20 auth_method 0 2023-09-30T00:48:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.497+0000 7fee41cb7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.497+0000 7fee41cb7700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.497+0000 7fee41cb7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.497+0000 7fee414b6700 10 monclient(hunting): get_auth_request con 0x7fee3c0a5fb0 auth_method 0 2023-09-30T00:48:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.497+0000 7fee414b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.497+0000 7fee414b6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.497+0000 7fee414b6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:01.503 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.498+0000 7fee41cb7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.498+0000 7fee41cb7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.498+0000 7fee41cb7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.498+0000 7fee414b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:01.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.498+0000 7fee414b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.498+0000 7fee414b6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.503+0000 7fee41cb7700 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2023-09-30T00:48:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.503+0000 7fee41cb7700 1 -- 172.21.15.40:0/3326160130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee3c0a5fb0 msgr2=0x7fee3c0aa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.503+0000 7fee41cb7700 1 --2- 172.21.15.40:0/3326160130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee3c0a5fb0 0x7fee3c0aa3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.503+0000 7fee41cb7700 1 -- 172.21.15.40:0/3326160130 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee3c0b1460 msgr2=0x7fee3c13e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.503+0000 7fee41cb7700 1 --2- 172.21.15.40:0/3326160130 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee3c0b1460 0x7fee3c13e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.503+0000 7fee41cb7700 10 monclient: _finish_hunting 0 2023-09-30T00:48:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.503+0000 7fee41cb7700 1 monclient: found mon.noname-a 2023-09-30T00:48:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.503+0000 7fee41cb7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.503+0000 7fee41cb7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:01.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.503+0000 7fee41cb7700 1 -- 172.21.15.40:0/3326160130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee34009ae0 con 0x7fee3c13ac20 2023-09-30T00:48:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.507+0000 7fee41cb7700 1 --2- 172.21.15.40:0/3326160130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee3c13ac20 0x7fee3c151120 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fee3400b6c0 tx=0x7fee34017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.507+0000 7fee317fa700 10 client.?.objecter ms_handle_connect 0x7fee3c13ac20 2023-09-30T00:48:01.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.507+0000 7fee317fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:01.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee317fa700 1 -- 172.21.15.40:0/3326160130 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3400be70 con 0x7fee3c13ac20 2023-09-30T00:48:01.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee317fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee317fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:01.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee317fa700 10 monclient: dump: 2023-09-30T00:48:01.523 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.523 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.524 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.524 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.524 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.524 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.524 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.524 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.524 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.524 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee317fa700 10 monclient: _finish_auth 0 2023-09-30T00:48:01.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee317fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:01.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee317fa700 1 -- 172.21.15.40:0/3326160130 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee340094e0 con 0x7fee3c13ac20 2023-09-30T00:48:01.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee4371a700 5 monclient: authenticate success, global_id 4353 2023-09-30T00:48:01.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee4371a700 10 monclient: _renew_subs 2023-09-30T00:48:01.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee4371a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:01.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee4371a700 1 -- 172.21.15.40:0/3326160130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3c0a8a50 con 0x7fee3c13ac20 2023-09-30T00:48:01.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee317fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:01.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee317fa700 1 -- 172.21.15.40:0/3326160130 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3401e430 con 0x7fee3c13ac20 2023-09-30T00:48:01.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee317fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee317fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:01.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee317fa700 10 monclient: dump: 2023-09-30T00:48:01.525 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.525 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.525 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.525 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.526 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.526 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.526 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.526 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.526 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.526 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee337fe700 4 set_mon_vals no callback set 2023-09-30T00:48:01.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee4371a700 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:01.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee4371a700 10 monclient: _renew_subs 2023-09-30T00:48:01.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee4371a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:01.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee4371a700 1 -- 172.21.15.40:0/3326160130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee3c0a9900 con 0x7fee3c13ac20 2023-09-30T00:48:01.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.522+0000 7fee4371a700 1 librados: init done 2023-09-30T00:48:01.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.523+0000 7fee4371a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:01.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.523+0000 7fee4371a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:01.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.523+0000 7fee4371a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:01.527 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.523+0000 7fee4371a700 1 -- 172.21.15.40:0/3326160130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3c0aa0c0 con 0x7fee3c13ac20 2023-09-30T00:48:01.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.588+0000 7f507b7fe700 1 -- 172.21.15.40:0/645373248 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f50740070d0 con 0x7f50840b1460 2023-09-30T00:48:01.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.589+0000 7f507b7fe700 1 --2- 172.21.15.40:0/645373248 >> v2:172.21.15.135:6800/116426 conn(0x7f5064068960 0x7f506406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.589+0000 7f507b7fe700 1 -- 172.21.15.40:0/645373248 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5074083de0 con 0x7f50840b1460 2023-09-30T00:48:01.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.589+0000 7f507b7fe700 10 client.4378.objecter ms_dispatch 0x7f5084001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:01.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.589+0000 7f507b7fe700 3 client.4378.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:01.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.589+0000 7f507b7fe700 3 client.4378.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:01.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.589+0000 7f507b7fe700 20 client.4378.objecter dump_active .. 0 homeless 2023-09-30T00:48:01.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.589+0000 7f5082ffd700 1 --2- 172.21.15.40:0/645373248 >> v2:172.21.15.135:6800/116426 conn(0x7f5064068960 0x7f506406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:01.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.589+0000 7f5082ffd700 10 monclient: get_auth_request con 0x7f5064068960 auth_method 0 2023-09-30T00:48:01.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.590+0000 7f507b7fe700 10 client.4378.objecter ms_handle_connect 0x7f5064068960 2023-09-30T00:48:01.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.590+0000 7f5082ffd700 1 --2- 172.21.15.40:0/645373248 >> v2:172.21.15.135:6800/116426 conn(0x7f5064068960 0x7f506406ae10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f506c0030c0 tx=0x7f506c016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:01.609 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:48:01.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.611+0000 7fdbd9335700 10 monclient: get_monmap_and_config 2023-09-30T00:48:01.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.611+0000 7fdbd9335700 10 monclient: build_initial_monmap 2023-09-30T00:48:01.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.611+0000 7fdbd9335700 10 monclient: monmap: 2023-09-30T00:48:01.613 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:01.613 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.613 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:01.612456+0000 2023-09-30T00:48:01.613 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:01.612456+0000 2023-09-30T00:48:01.613 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:01.613 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.613 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:01.613 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:01.613 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:01.613 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:01.614 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:01.614 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 1 Processor -- start 2023-09-30T00:48:01.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 1 -- start start 2023-09-30T00:48:01.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 10 monclient: init 2023-09-30T00:48:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-09-30T00:48:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbd40a5710 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdbd413e1d0 addr v1:172.21.15.40:6789/0 2023-09-30T00:48:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdbd4146d00 0x7fdbd41490e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 10 monclient(hunting): picked mon.noname-a con 0x7fdbd4146d00 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd40a7ff0 con 0x7fdbd413e1d0 2023-09-30T00:48:01.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd40a7cf0 con 0x7fdbd40a5710 2023-09-30T00:48:01.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbd40aedc0 con 0x7fdbd4146d00 2023-09-30T00:48:01.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:01.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.612+0000 7fdbd9335700 10 monclient(hunting): authenticate will time out at 5342.913934s 2023-09-30T00:48:01.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.634+0000 7fdbd2ffd700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fdbd40a5710 0x7fdbd40a75e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:58102/0 (socket says 172.21.15.40:58102) 2023-09-30T00:48:01.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.634+0000 7fdbd2ffd700 1 -- 172.21.15.40:0/1267512941 learned_addr learned my addr 172.21.15.40:0/1267512941 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:48:01.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.634+0000 7f507b7fe700 1 -- 172.21.15.40:0/645373248 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5074051700 con 0x7f50840b1460 2023-09-30T00:48:01.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.634+0000 7f507b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:01.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.634+0000 7f507b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:01.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.648+0000 7fee317fa700 1 -- 172.21.15.40:0/3326160130 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fee3401e880 con 0x7fee3c13ac20 2023-09-30T00:48:01.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.648+0000 7fee317fa700 1 --2- 172.21.15.40:0/3326160130 >> v2:172.21.15.135:6800/116426 conn(0x7fee20068890 0x7fee2006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.648+0000 7fee317fa700 1 -- 172.21.15.40:0/3326160130 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fee34086470 con 0x7fee3c13ac20 2023-09-30T00:48:01.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.648+0000 7fee317fa700 10 client.4353.objecter ms_dispatch 0x7fee3c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:01.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.648+0000 7fee317fa700 3 client.4353.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:01.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.648+0000 7fee317fa700 3 client.4353.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:01.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.648+0000 7fee317fa700 20 client.4353.objecter dump_active .. 0 homeless 2023-09-30T00:48:01.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.649+0000 7fee414b6700 1 --2- 172.21.15.40:0/3326160130 >> v2:172.21.15.135:6800/116426 conn(0x7fee20068890 0x7fee2006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:01.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.649+0000 7fee414b6700 10 monclient: get_auth_request con 0x7fee20068890 auth_method 0 2023-09-30T00:48:01.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.649+0000 7fee414b6700 1 --2- 172.21.15.40:0/3326160130 >> v2:172.21.15.135:6800/116426 conn(0x7fee20068890 0x7fee2006ad40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fee38009bf0 tx=0x7fee380091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:01.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.649+0000 7fee317fa700 10 client.4353.objecter ms_handle_connect 0x7fee20068890 2023-09-30T00:48:01.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.743+0000 7f10c0d5c700 10 monclient: get_monmap_and_config 2023-09-30T00:48:01.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.743+0000 7f10c0d5c700 10 monclient: build_initial_monmap 2023-09-30T00:48:01.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.743+0000 7f10c0d5c700 10 monclient: monmap: 2023-09-30T00:48:01.744 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:01.744 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.744 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:01.744717+0000 2023-09-30T00:48:01.745 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:01.744717+0000 2023-09-30T00:48:01.745 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:01.745 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.745 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:01.745 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:01.745 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:01.745 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:01.745 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:01.745 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.744+0000 7f10c0d5c700 1 Processor -- start 2023-09-30T00:48:01.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.744+0000 7f10c0d5c700 1 -- start start 2023-09-30T00:48:01.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.744+0000 7f10c0d5c700 10 monclient: init 2023-09-30T00:48:01.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.745+0000 7f10c0d5c700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:01.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.745+0000 7f10c0d5c700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-09-30T00:48:01.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.745+0000 7f10c0d5c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f10bc0b1460 0x7f10bc0b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.745+0000 7f10c0d5c700 10 monclient(hunting): picked mon.noname-a con 0x7f10bc0b1460 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:01.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.745+0000 7f10c0d5c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10bc0a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:01.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.745+0000 7f10c0d5c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10bc13ac20 0x7f10bc13b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.745+0000 7f10c0d5c700 10 monclient(hunting): picked mon.noname-b con 0x7f10bc13ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.745+0000 7f10c0d5c700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10bc0a5b20 con 0x7f10bc0a5fb0 2023-09-30T00:48:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.745+0000 7f10c0d5c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.745+0000 7f10c0d5c700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10bc0a71f0 con 0x7f10bc0b1460 2023-09-30T00:48:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.745+0000 7f10c0d5c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.745+0000 7f10c0d5c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0a74d0 con 0x7f10bc13ac20 2023-09-30T00:48:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.745+0000 7f10c0d5c700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.745+0000 7f10c0d5c700 10 monclient(hunting): authenticate will time out at 5343.046368s 2023-09-30T00:48:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.746+0000 7f10bad9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10bc13ac20 0x7f10bc13b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:01.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.746+0000 7f10b9d9b700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f10bc0a5fb0 0x7f10bc141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:58118/0 (socket says 172.21.15.40:58118) 2023-09-30T00:48:01.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.746+0000 7f10b9d9b700 1 -- 172.21.15.40:0/99437673 learned_addr learned my addr 172.21.15.40:0/99437673 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:48:01.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.746+0000 7f10bad9d700 10 monclient(hunting): get_auth_request con 0x7f10bc13ac20 auth_method 0 2023-09-30T00:48:01.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.746+0000 7f10bad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:01.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.746+0000 7f10bad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:01.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.746+0000 7f10bad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:01.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.747+0000 7f10bad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:01.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.747+0000 7f10bad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:01.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.747+0000 7f10bad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:01.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.751+0000 7f10bad9d700 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2023-09-30T00:48:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.751+0000 7f10bad9d700 1 -- 172.21.15.40:0/99437673 >> v2:172.21.15.40:3300/0 conn(0x7f10bc0b1460 msgr2=0x7f10bc0b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.751+0000 7f10bad9d700 1 --2- 172.21.15.40:0/99437673 >> v2:172.21.15.40:3300/0 conn(0x7f10bc0b1460 0x7f10bc0b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.751+0000 7f10bad9d700 1 -- 172.21.15.40:0/99437673 >> v1:172.21.15.135:6789/0 conn(0x7f10bc0a5fb0 legacy=0x7f10bc141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.751+0000 7f10bad9d700 10 monclient: _finish_hunting 0 2023-09-30T00:48:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.751+0000 7f10bad9d700 1 monclient: found mon.noname-b 2023-09-30T00:48:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.751+0000 7f10bad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.751+0000 7f10bad9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:48:01.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.751+0000 7f10bad9d700 1 -- 172.21.15.40:0/99437673 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10bc0a63c0 con 0x7f10bc13ac20 2023-09-30T00:48:01.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.754+0000 7f10bad9d700 1 --2- 172.21.15.40:0/99437673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10bc13ac20 0x7f10bc13b080 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f10b40098e0 tx=0x7f10b40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16bacbc4a8450eea server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10b959a700 1 -- 172.21.15.40:0/99437673 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b4009e80 con 0x7f10bc13ac20 2023-09-30T00:48:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10b959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10b959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10b959a700 10 monclient: dump: 2023-09-30T00:48:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.762 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.763 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10b959a700 10 monclient: _finish_auth 0 2023-09-30T00:48:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10b959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10b959a700 1 -- 172.21.15.40:0/99437673 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b400b860 con 0x7f10bc13ac20 2023-09-30T00:48:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10c0d5c700 5 monclient: authenticate success, global_id 4319 2023-09-30T00:48:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10c0d5c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10b959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:01.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10c0d5c700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10b959a700 1 -- 172.21.15.40:0/99437673 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b400bbe0 con 0x7f10bc13ac20 2023-09-30T00:48:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10c0d5c700 4 set_mon_vals no callback set 2023-09-30T00:48:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10b959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10b959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.760+0000 7f10b959a700 10 monclient: dump: 2023-09-30T00:48:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.764 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.765 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.761+0000 7f10c0d5c700 10 monclient: shutdown 2023-09-30T00:48:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.761+0000 7f10c0d5c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.761+0000 7f10c0d5c700 1 -- 172.21.15.40:0/99437673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10bc13ac20 msgr2=0x7f10bc13b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.761+0000 7f10c0d5c700 1 --2- 172.21.15.40:0/99437673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10bc13ac20 0x7f10bc13b080 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f10b40098e0 tx=0x7f10b40091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.761+0000 7f10c0d5c700 1 -- 172.21.15.40:0/99437673 shutdown_connections 2023-09-30T00:48:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.761+0000 7f10c0d5c700 1 --2- 172.21.15.40:0/99437673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10bc13ac20 0x7f10bc13b080 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.761+0000 7f10c0d5c700 1 --2- 172.21.15.40:0/99437673 >> v2:172.21.15.40:3300/0 conn(0x7f10bc0b1460 0x7f10bc0b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.761+0000 7f10c0d5c700 1 -- 172.21.15.40:0/99437673 >> 172.21.15.40:0/99437673 conn(0x7f10bc0631f0 msgr2=0x7f10bc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:01.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.761+0000 7f10c0d5c700 1 -- 172.21.15.40:0/99437673 shutdown_connections 2023-09-30T00:48:01.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.761+0000 7f10c0d5c700 1 -- 172.21.15.40:0/99437673 wait complete. 2023-09-30T00:48:01.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.761+0000 7f10c0d5c700 10 monclient: build_initial_monmap 2023-09-30T00:48:01.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.761+0000 7f10c0d5c700 10 monclient: monmap: 2023-09-30T00:48:01.768 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:01.768 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:01.768 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:01.768 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:01.768 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:01.768 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:01.769 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:01.769 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:01.769 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:01.769 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.762+0000 7f10c0d5c700 1 librados: starting msgr at 2023-09-30T00:48:01.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.762+0000 7f10c0d5c700 1 librados: starting objecter 2023-09-30T00:48:01.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.762+0000 7f10c0d5c700 1 Processor -- start 2023-09-30T00:48:01.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.762+0000 7f10c0d5c700 1 -- start start 2023-09-30T00:48:01.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.762+0000 7f10c0d5c700 1 librados: setting wanted keys 2023-09-30T00:48:01.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.762+0000 7f10c0d5c700 1 librados: calling monclient init 2023-09-30T00:48:01.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.762+0000 7f10c0d5c700 10 monclient: init 2023-09-30T00:48:01.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:01.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:48:01.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10bc0a5fb0 0x7f10bc13e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 10 monclient(hunting): picked mon.noname-b con 0x7f10bc0a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:01.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10bc0b1460 0x7f10bc1516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 10 monclient(hunting): picked mon.noname-a con 0x7f10bc0b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:01.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10bc13ac20 0x7f10bc0a94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 10 monclient(hunting): picked mon.noname-c con 0x7f10bc13ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:01.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc152120 con 0x7f10bc0b1460 2023-09-30T00:48:01.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0ab480 con 0x7f10bc0a5fb0 2023-09-30T00:48:01.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc151c30 con 0x7f10bc13ac20 2023-09-30T00:48:01.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:01.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10c0d5c700 10 monclient(hunting): authenticate will time out at 5343.064153s 2023-09-30T00:48:01.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10ba59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10bc0a5fb0 0x7f10bc13e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:01.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10ba59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10bc0a5fb0 0x7f10bc13e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:41444/0 (socket says 172.21.15.40:41444) 2023-09-30T00:48:01.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10ba59c700 1 -- 172.21.15.40:0/886761637 learned_addr learned my addr 172.21.15.40:0/886761637 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:01.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10ba59c700 10 monclient(hunting): get_auth_request con 0x7f10bc0a5fb0 auth_method 0 2023-09-30T00:48:01.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10ba59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:01.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10ba59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:01.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.763+0000 7f10ba59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.764+0000 7f10ba59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.764+0000 7f10ba59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.764+0000 7f10ba59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.764+0000 7f10bad9d700 1 --2- 172.21.15.40:0/886761637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10bc13ac20 0x7f10bc0a94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.764+0000 7f10bad9d700 10 monclient(hunting): get_auth_request con 0x7f10bc13ac20 auth_method 0 2023-09-30T00:48:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.764+0000 7f10bad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.764+0000 7f10bad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.764+0000 7f10bad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.765+0000 7f10bad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.765+0000 7f10bad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.765+0000 7f10bad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.767+0000 7f10ba59c700 10 monclient(hunting): handle_auth_done global_id 4381 payload 824 2023-09-30T00:48:01.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.767+0000 7f10ba59c700 1 -- 172.21.15.40:0/886761637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10bc13ac20 msgr2=0x7f10bc0a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.767+0000 7f10ba59c700 1 --2- 172.21.15.40:0/886761637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10bc13ac20 0x7f10bc0a94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.767+0000 7f10ba59c700 1 -- 172.21.15.40:0/886761637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10bc0b1460 msgr2=0x7f10bc1516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.767+0000 7f10ba59c700 1 --2- 172.21.15.40:0/886761637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10bc0b1460 0x7f10bc1516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.767+0000 7f10ba59c700 10 monclient: _finish_hunting 0 2023-09-30T00:48:01.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.767+0000 7f10ba59c700 1 monclient: found mon.noname-b 2023-09-30T00:48:01.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.767+0000 7f10ba59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:01.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.767+0000 7f10ba59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:01.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.767+0000 7f10ba59c700 1 -- 172.21.15.40:0/886761637 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b4009420 con 0x7f10bc0a5fb0 2023-09-30T00:48:01.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.770+0000 7f10ba59c700 1 --2- 172.21.15.40:0/886761637 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10bc0a5fb0 0x7f10bc13e5f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00bc30 tx=0x7f10ac009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:01.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.770+0000 7f10aa7fc700 10 client.?.objecter ms_handle_connect 0x7f10bc0a5fb0 2023-09-30T00:48:01.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.770+0000 7f10aa7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:01.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10aa7fc700 1 -- 172.21.15.40:0/886761637 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac0092e0 con 0x7f10bc0a5fb0 2023-09-30T00:48:01.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10aa7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:01.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10aa7fc700 10 monclient: dump: 2023-09-30T00:48:01.775 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.776 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.776 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.776 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.776 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.776 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.776 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.776 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.776 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.776 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10aa7fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:01.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:01.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10aa7fc700 1 -- 172.21.15.40:0/886761637 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ac00c6f0 con 0x7f10bc0a5fb0 2023-09-30T00:48:01.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10c0d5c700 5 monclient: authenticate success, global_id 4381 2023-09-30T00:48:01.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10c0d5c700 10 monclient: _renew_subs 2023-09-30T00:48:01.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10c0d5c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:01.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10c0d5c700 1 -- 172.21.15.40:0/886761637 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10bc151df0 con 0x7f10bc0a5fb0 2023-09-30T00:48:01.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10aa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:01.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10aa7fc700 1 -- 172.21.15.40:0/886761637 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac00cad0 con 0x7f10bc0a5fb0 2023-09-30T00:48:01.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10aa7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:01.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10aa7fc700 10 monclient: dump: 2023-09-30T00:48:01.777 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.777 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.777 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.778 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10b8d99700 4 set_mon_vals no callback set 2023-09-30T00:48:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10c0d5c700 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10c0d5c700 10 monclient: _renew_subs 2023-09-30T00:48:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10c0d5c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10c0d5c700 1 -- 172.21.15.40:0/886761637 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10bc0a9c60 con 0x7f10bc0a5fb0 2023-09-30T00:48:01.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.773+0000 7f10c0d5c700 1 librados: init done 2023-09-30T00:48:01.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.774+0000 7f10c0d5c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:01.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.774+0000 7f10c0d5c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:01.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.774+0000 7f10c0d5c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:01.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.774+0000 7f10c0d5c700 1 -- 172.21.15.40:0/886761637 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10bc064a00 con 0x7f10bc0a5fb0 2023-09-30T00:48:01.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.868+0000 7f508a0b2700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T00:48:01.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.868+0000 7f508a0b2700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T00:48:01.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.868+0000 7f508a0b2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:01.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.868+0000 7f508a0b2700 1 -- 172.21.15.40:0/645373248 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f50840aac30 con 0x7f50840b1460 2023-09-30T00:48:01.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.927+0000 7fee317fa700 1 -- 172.21.15.40:0/3326160130 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fee34053d90 con 0x7fee3c13ac20 2023-09-30T00:48:01.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.927+0000 7fee317fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:01.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.927+0000 7fee317fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:01.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.928+0000 7fdbd37fe700 1 --2- 172.21.15.40:0/1267512941 >> v2:172.21.15.40:3300/0 conn(0x7fdbd4146d00 0x7fdbd41490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:01.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.929+0000 7fdbd37fe700 10 monclient(hunting): get_auth_request con 0x7fdbd4146d00 auth_method 0 2023-09-30T00:48:01.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.929+0000 7fdbd37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:01.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.929+0000 7fdbd37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:01.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.929+0000 7fdbd37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:01.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.932+0000 7fdbd37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:01.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.932+0000 7fdbd37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:01.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.932+0000 7fdbd37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:01.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.935+0000 7fdbd37fe700 10 monclient(hunting): handle_auth_done global_id 4356 payload 274 2023-09-30T00:48:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.935+0000 7fdbd37fe700 1 -- 172.21.15.40:0/1267512941 >> v1:172.21.15.135:6789/0 conn(0x7fdbd40a5710 legacy=0x7fdbd40a75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.935+0000 7fdbd37fe700 1 -- 172.21.15.40:0/1267512941 >> v1:172.21.15.40:6789/0 conn(0x7fdbd413e1d0 legacy=0x7fdbd413e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.935+0000 7fdbd37fe700 10 monclient: _finish_hunting 0 2023-09-30T00:48:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.935+0000 7fdbd37fe700 1 monclient: found mon.noname-a 2023-09-30T00:48:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.935+0000 7fdbd37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.935+0000 7fdbd37fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.935+0000 7fdbd37fe700 1 -- 172.21.15.40:0/1267512941 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd40aef00 con 0x7fdbd4146d00 2023-09-30T00:48:01.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.939+0000 7fdbd37fe700 1 --2- 172.21.15.40:0/1267512941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd4146d00 0x7fdbd41490e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc009e60 tx=0x7fdbcc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e18c1b75e494611 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:01.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd1ffb700 1 -- 172.21.15.40:0/1267512941 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc00e070 con 0x7fdbd4146d00 2023-09-30T00:48:01.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:01.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd1ffb700 10 monclient: dump: 2023-09-30T00:48:01.950 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.950 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.950 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.950 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.950 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.950 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.951 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.951 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.951 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.951 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd1ffb700 10 monclient: _finish_auth 0 2023-09-30T00:48:01.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:01.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd1ffb700 1 -- 172.21.15.40:0/1267512941 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbcc00bc30 con 0x7fdbd4146d00 2023-09-30T00:48:01.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd9335700 5 monclient: authenticate success, global_id 4356 2023-09-30T00:48:01.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd9335700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:01.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:01.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd1ffb700 1 -- 172.21.15.40:0/1267512941 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbcc012430 con 0x7fdbd4146d00 2023-09-30T00:48:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd9335700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd9335700 4 set_mon_vals no callback set 2023-09-30T00:48:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.949+0000 7fdbd1ffb700 10 monclient: dump: 2023-09-30T00:48:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.952 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.953 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.953 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.950+0000 7fdbd9335700 10 monclient: shutdown 2023-09-30T00:48:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.950+0000 7fdbd9335700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.950+0000 7fdbd9335700 1 -- 172.21.15.40:0/1267512941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd4146d00 msgr2=0x7fdbd41490e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.950+0000 7fdbd9335700 1 --2- 172.21.15.40:0/1267512941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd4146d00 0x7fdbd41490e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc009e60 tx=0x7fdbcc009780 comp rx=0 tx=0).stop 2023-09-30T00:48:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.950+0000 7fdbd9335700 1 -- 172.21.15.40:0/1267512941 shutdown_connections 2023-09-30T00:48:01.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.950+0000 7fdbd9335700 1 --2- 172.21.15.40:0/1267512941 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd4146d00 0x7fdbd41490e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.950+0000 7fdbd9335700 1 -- 172.21.15.40:0/1267512941 >> 172.21.15.40:0/1267512941 conn(0x7fdbd4069080 msgr2=0x7fdbd40631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.950+0000 7fdbd9335700 1 -- 172.21.15.40:0/1267512941 shutdown_connections 2023-09-30T00:48:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.950+0000 7fdbd9335700 1 -- 172.21.15.40:0/1267512941 wait complete. 2023-09-30T00:48:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.951+0000 7fdbd9335700 10 monclient: build_initial_monmap 2023-09-30T00:48:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.951+0000 7fdbd9335700 10 monclient: monmap: 2023-09-30T00:48:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:01.956 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:01.957 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.951+0000 7fdbd9335700 1 librados: starting msgr at 2023-09-30T00:48:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.951+0000 7fdbd9335700 1 librados: starting objecter 2023-09-30T00:48:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.951+0000 7fdbd9335700 1 Processor -- start 2023-09-30T00:48:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.951+0000 7fdbd9335700 1 -- start start 2023-09-30T00:48:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.951+0000 7fdbd9335700 1 librados: setting wanted keys 2023-09-30T00:48:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.951+0000 7fdbd9335700 1 librados: calling monclient init 2023-09-30T00:48:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.951+0000 7fdbd9335700 10 monclient: init 2023-09-30T00:48:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:01.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:48:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbd40a5710 0x7fdbd40a98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 10 monclient(hunting): picked mon.noname-c con 0x7fdbd40a5710 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd413e1d0 0x7fdbd40abb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 10 monclient(hunting): picked mon.noname-a con 0x7fdbd413e1d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdbd4146d00 0x7fdbd40ae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 10 monclient(hunting): picked mon.noname-b con 0x7fdbd4146d00 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd40ada60 con 0x7fdbd413e1d0 2023-09-30T00:48:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd40add40 con 0x7fdbd4146d00 2023-09-30T00:48:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:01.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd40ab320 con 0x7fdbd40a5710 2023-09-30T00:48:01.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:01.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.952+0000 7fdbd9335700 10 monclient(hunting): authenticate will time out at 5343.253431s 2023-09-30T00:48:01.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.953+0000 7fdbd27fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd413e1d0 0x7fdbd40abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:01.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.953+0000 7fdbd27fc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd413e1d0 0x7fdbd40abb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52386/0 (socket says 172.21.15.40:52386) 2023-09-30T00:48:01.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.953+0000 7fdbd27fc700 1 -- 172.21.15.40:0/1844732692 learned_addr learned my addr 172.21.15.40:0/1844732692 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:01.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.953+0000 7fdbd27fc700 10 monclient(hunting): get_auth_request con 0x7fdbd413e1d0 auth_method 0 2023-09-30T00:48:01.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.953+0000 7fdbd27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:01.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.953+0000 7fdbd27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:01.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.953+0000 7fdbd27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:01.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.954+0000 7fdbd27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:01.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.954+0000 7fdbd27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:01.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.954+0000 7fdbd27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:01.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.958+0000 7fdbd27fc700 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2023-09-30T00:48:01.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.958+0000 7fdbd27fc700 1 -- 172.21.15.40:0/1844732692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbd40a5710 msgr2=0x7fdbd40a98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.958+0000 7fdbd27fc700 1 --2- 172.21.15.40:0/1844732692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbd40a5710 0x7fdbd40a98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.958+0000 7fdbd27fc700 1 -- 172.21.15.40:0/1844732692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdbd4146d00 msgr2=0x7fdbd40ae4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:01.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.958+0000 7fdbd27fc700 1 --2- 172.21.15.40:0/1844732692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdbd4146d00 0x7fdbd40ae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:01.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.958+0000 7fdbd27fc700 10 monclient: _finish_hunting 0 2023-09-30T00:48:01.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.958+0000 7fdbd27fc700 1 monclient: found mon.noname-a 2023-09-30T00:48:01.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.958+0000 7fdbd27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:01.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.958+0000 7fdbd27fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:01.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.958+0000 7fdbd27fc700 1 -- 172.21.15.40:0/1844732692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbcc009a20 con 0x7fdbd413e1d0 2023-09-30T00:48:01.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.962+0000 7fdbd27fc700 1 --2- 172.21.15.40:0/1844732692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd413e1d0 0x7fdbd40abb80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800c790 tx=0x7fdbc8007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:01.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.962+0000 7fdbbaffd700 10 client.?.objecter ms_handle_connect 0x7fdbd413e1d0 2023-09-30T00:48:01.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.962+0000 7fdbbaffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:01.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbbaffd700 1 -- 172.21.15.40:0/1844732692 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8011590 con 0x7fdbd413e1d0 2023-09-30T00:48:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbbaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbbaffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbbaffd700 10 monclient: dump: 2023-09-30T00:48:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.968 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.969 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.969 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.969 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbbaffd700 10 monclient: _finish_auth 0 2023-09-30T00:48:01.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbbaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:01.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbbaffd700 1 -- 172.21.15.40:0/1844732692 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc8011710 con 0x7fdbd413e1d0 2023-09-30T00:48:01.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbd9335700 5 monclient: authenticate success, global_id 4359 2023-09-30T00:48:01.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbd9335700 10 monclient: _renew_subs 2023-09-30T00:48:01.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbd9335700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:01.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbd9335700 1 -- 172.21.15.40:0/1844732692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd40aa540 con 0x7fdbd413e1d0 2023-09-30T00:48:01.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbbaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:01.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbbaffd700 1 -- 172.21.15.40:0/1844732692 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc8011ad0 con 0x7fdbd413e1d0 2023-09-30T00:48:01.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbbaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:01.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbbaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:01.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbbaffd700 10 monclient: dump: 2023-09-30T00:48:01.970 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:01.970 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:01.970 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.970 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:01.970 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:01.971 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:01.971 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:01.971 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:01.971 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:01.971 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:01.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbd17fa700 4 set_mon_vals no callback set 2023-09-30T00:48:01.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbd9335700 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:01.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbd9335700 10 monclient: _renew_subs 2023-09-30T00:48:01.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbd9335700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:01.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbd9335700 1 -- 172.21.15.40:0/1844732692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd40ac280 con 0x7fdbd413e1d0 2023-09-30T00:48:01.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.967+0000 7fdbd9335700 1 librados: init done 2023-09-30T00:48:01.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.968+0000 7fdbd9335700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:01.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.968+0000 7fdbd9335700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:01.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.968+0000 7fdbd9335700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:01.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:01.968+0000 7fdbd9335700 1 -- 172.21.15.40:0/1844732692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd40ab180 con 0x7fdbd413e1d0 2023-09-30T00:48:02.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.003+0000 7fdbbaffd700 1 -- 172.21.15.40:0/1844732692 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fdbc801b3b0 con 0x7fdbd413e1d0 2023-09-30T00:48:02.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.004+0000 7fdbbaffd700 1 --2- 172.21.15.40:0/1844732692 >> v2:172.21.15.135:6800/116426 conn(0x7fdbc0068910 0x7fdbc006adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:02.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.004+0000 7fdbbaffd700 1 -- 172.21.15.40:0/1844732692 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fdbc8083490 con 0x7fdbd413e1d0 2023-09-30T00:48:02.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.004+0000 7fdbbaffd700 10 client.4359.objecter ms_dispatch 0x7fdbd4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:02.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.004+0000 7fdbbaffd700 3 client.4359.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:02.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.004+0000 7fdbbaffd700 3 client.4359.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:02.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.004+0000 7fdbbaffd700 20 client.4359.objecter dump_active .. 0 homeless 2023-09-30T00:48:02.006 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.004+0000 7fdbd2ffd700 1 --2- 172.21.15.40:0/1844732692 >> v2:172.21.15.135:6800/116426 conn(0x7fdbc0068910 0x7fdbc006adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:02.006 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.004+0000 7fdbd2ffd700 10 monclient: get_auth_request con 0x7fdbc0068910 auth_method 0 2023-09-30T00:48:02.006 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.004+0000 7fdbbaffd700 10 client.4359.objecter ms_handle_connect 0x7fdbc0068910 2023-09-30T00:48:02.006 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.004+0000 7fdbd2ffd700 1 --2- 172.21.15.40:0/1844732692 >> v2:172.21.15.135:6800/116426 conn(0x7fdbc0068910 0x7fdbc006adc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40016a0 tx=0x7fdbc4007b30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:02.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.027+0000 7f10aa7fc700 1 -- 172.21.15.40:0/886761637 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f10ac012070 con 0x7f10bc0a5fb0 2023-09-30T00:48:02.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.027+0000 7f10aa7fc700 1 --2- 172.21.15.40:0/886761637 >> v2:172.21.15.135:6800/116426 conn(0x7f109c068960 0x7f109c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:02.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.027+0000 7f10aa7fc700 1 -- 172.21.15.40:0/886761637 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f10ac083d90 con 0x7f10bc0a5fb0 2023-09-30T00:48:02.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.027+0000 7f10aa7fc700 10 client.4381.objecter ms_dispatch 0x7f10bc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:02.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.027+0000 7f10aa7fc700 3 client.4381.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:02.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.027+0000 7f10aa7fc700 3 client.4381.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:02.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.027+0000 7f10aa7fc700 20 client.4381.objecter dump_active .. 0 homeless 2023-09-30T00:48:02.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.027+0000 7f10b9d9b700 1 --2- 172.21.15.40:0/886761637 >> v2:172.21.15.135:6800/116426 conn(0x7f109c068960 0x7f109c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:02.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.028+0000 7f10b9d9b700 10 monclient: get_auth_request con 0x7f109c068960 auth_method 0 2023-09-30T00:48:02.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.028+0000 7f10b9d9b700 1 --2- 172.21.15.40:0/886761637 >> v2:172.21.15.135:6800/116426 conn(0x7f109c068960 0x7f109c06ae10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f10bc13f600 tx=0x7f10b0007b30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:02.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.028+0000 7f10aa7fc700 10 client.4381.objecter ms_handle_connect 0x7f109c068960 2023-09-30T00:48:02.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.036+0000 7f10aa7fc700 1 -- 172.21.15.40:0/886761637 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f10ac051760 con 0x7f10bc0a5fb0 2023-09-30T00:48:02.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.036+0000 7f10aa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:02.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.036+0000 7f10aa7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.112+0000 7f507b7fe700 1 -- 172.21.15.40:0/645373248 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5074021090 con 0x7f50840b1460 2023-09-30T00:48:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.112+0000 7f507b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T00:48:02.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.112+0000 7f507b7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:02.113 INFO:teuthology.orchestra.run.smithi040.stdout:64424509446 2023-09-30T00:48:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.114+0000 7f508a0b2700 10 librados: watch_flush enter 2023-09-30T00:48:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.114+0000 7f508a0b2700 10 librados: watch_flush exit 2023-09-30T00:48:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.114+0000 7f508a0b2700 20 client.4378.objecter shutdown clearing up homeless session... 2023-09-30T00:48:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.114+0000 7f508a0b2700 10 client.4378.objecter successfully canceled tick 2023-09-30T00:48:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.114+0000 7f508a0b2700 1 -- 172.21.15.40:0/645373248 >> v2:172.21.15.135:6800/116426 conn(0x7f5064068960 msgr2=0x7f506406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.115+0000 7f508a0b2700 1 --2- 172.21.15.40:0/645373248 >> v2:172.21.15.135:6800/116426 conn(0x7f5064068960 0x7f506406ae10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f506c0030c0 tx=0x7f506c016000 comp rx=0 tx=0).stop 2023-09-30T00:48:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.115+0000 7f508a0b2700 10 monclient: shutdown 2023-09-30T00:48:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.115+0000 7f508a0b2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.115+0000 7f508a0b2700 1 -- 172.21.15.40:0/645373248 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50840b1460 msgr2=0x7f50840a8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.115+0000 7f508a0b2700 1 --2- 172.21.15.40:0/645373248 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50840b1460 0x7f50840a8d30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f507400c7c0 tx=0x7f5074007f70 comp rx=0 tx=0).stop 2023-09-30T00:48:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.115+0000 7f508a0b2700 1 -- 172.21.15.40:0/645373248 shutdown_connections 2023-09-30T00:48:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.115+0000 7f508a0b2700 1 --2- 172.21.15.40:0/645373248 >> v2:172.21.15.135:6800/116426 conn(0x7f5064068960 0x7f506406ae10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.115+0000 7f508a0b2700 1 --2- 172.21.15.40:0/645373248 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f50840b1460 0x7f50840a8d30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.115+0000 7f508a0b2700 1 --2- 172.21.15.40:0/645373248 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f508413ac20 0x7f508413d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.115+0000 7f508a0b2700 1 --2- 172.21.15.40:0/645373248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f50840b1830 0x7f50840a86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.115+0000 7f508a0b2700 1 -- 172.21.15.40:0/645373248 >> 172.21.15.40:0/645373248 conn(0x7f50840631f0 msgr2=0x7f50840b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.115+0000 7f508a0b2700 1 -- 172.21.15.40:0/645373248 shutdown_connections 2023-09-30T00:48:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.115+0000 7f508a0b2700 1 -- 172.21.15.40:0/645373248 wait complete. 2023-09-30T00:48:02.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.115+0000 7f508a0b2700 1 librados: shutdown 2023-09-30T00:48:02.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.7 2023-09-30T00:48:02.130 DEBUG:teuthology.parallel:result is None 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.140+0000 7fee4371a700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.140+0000 7fee4371a700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.140+0000 7fee4371a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.140+0000 7fee4371a700 1 -- 172.21.15.40:0/3326160130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fee3c064a00 con 0x7fee3c13ac20 2023-09-30T00:48:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.242+0000 7f10c0d5c700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:48:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.242+0000 7f10c0d5c700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:48:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.242+0000 7f10c0d5c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.242+0000 7f10c0d5c700 1 -- 172.21.15.40:0/886761637 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f10bc0a56d0 con 0x7f10bc0a5fb0 2023-09-30T00:48:02.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.247+0000 7f10aa7fc700 1 -- 172.21.15.40:0/886761637 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f10ac054d80 con 0x7f10bc0a5fb0 2023-09-30T00:48:02.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.247+0000 7f10aa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:48:02.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.247+0000 7f10aa7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:02.248 INFO:teuthology.orchestra.run.smithi040.stdout:85899345927 2023-09-30T00:48:02.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 10 librados: watch_flush enter 2023-09-30T00:48:02.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 10 librados: watch_flush exit 2023-09-30T00:48:02.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 20 client.4381.objecter shutdown clearing up homeless session... 2023-09-30T00:48:02.250 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 10 client.4381.objecter successfully canceled tick 2023-09-30T00:48:02.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 1 -- 172.21.15.40:0/886761637 >> v2:172.21.15.135:6800/116426 conn(0x7f109c068960 msgr2=0x7f109c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 1 --2- 172.21.15.40:0/886761637 >> v2:172.21.15.135:6800/116426 conn(0x7f109c068960 0x7f109c06ae10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f10bc13f600 tx=0x7f10b0007b30 comp rx=0 tx=0).stop 2023-09-30T00:48:02.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 10 monclient: shutdown 2023-09-30T00:48:02.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:02.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 1 -- 172.21.15.40:0/886761637 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10bc0a5fb0 msgr2=0x7f10bc13e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 1 --2- 172.21.15.40:0/886761637 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10bc0a5fb0 0x7f10bc13e5f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00bc30 tx=0x7f10ac009840 comp rx=0 tx=0).stop 2023-09-30T00:48:02.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 1 -- 172.21.15.40:0/886761637 shutdown_connections 2023-09-30T00:48:02.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 1 --2- 172.21.15.40:0/886761637 >> v2:172.21.15.135:6800/116426 conn(0x7f109c068960 0x7f109c06ae10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 1 --2- 172.21.15.40:0/886761637 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10bc0a5fb0 0x7f10bc13e5f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 1 --2- 172.21.15.40:0/886761637 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10bc13ac20 0x7f10bc0a94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 1 --2- 172.21.15.40:0/886761637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10bc0b1460 0x7f10bc1516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 1 -- 172.21.15.40:0/886761637 >> 172.21.15.40:0/886761637 conn(0x7f10bc0631f0 msgr2=0x7f10bc0afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:02.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 1 -- 172.21.15.40:0/886761637 shutdown_connections 2023-09-30T00:48:02.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 1 -- 172.21.15.40:0/886761637 wait complete. 2023-09-30T00:48:02.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.249+0000 7f10c0d5c700 1 librados: shutdown 2023-09-30T00:48:02.263 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345927 got 85899345927 for osd.0 2023-09-30T00:48:02.264 DEBUG:teuthology.parallel:result is None 2023-09-30T00:48:02.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.314+0000 7fdbbaffd700 1 -- 172.21.15.40:0/1844732692 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdbc8050d30 con 0x7fdbd413e1d0 2023-09-30T00:48:02.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.314+0000 7fdbbaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:02.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.314+0000 7fdbbaffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:02.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.326+0000 7fee317fa700 1 -- 172.21.15.40:0/3326160130 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fee340573b0 con 0x7fee3c13ac20 2023-09-30T00:48:02.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.326+0000 7fee317fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:48:02.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.326+0000 7fee317fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:02.327 INFO:teuthology.orchestra.run.smithi040.stdout:64424509447 2023-09-30T00:48:02.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 10 librados: watch_flush enter 2023-09-30T00:48:02.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 10 librados: watch_flush exit 2023-09-30T00:48:02.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 20 client.4353.objecter shutdown clearing up homeless session... 2023-09-30T00:48:02.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 10 client.4353.objecter successfully canceled tick 2023-09-30T00:48:02.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 1 -- 172.21.15.40:0/3326160130 >> v2:172.21.15.135:6800/116426 conn(0x7fee20068890 msgr2=0x7fee2006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 1 --2- 172.21.15.40:0/3326160130 >> v2:172.21.15.135:6800/116426 conn(0x7fee20068890 0x7fee2006ad40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fee38009bf0 tx=0x7fee380091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 10 monclient: shutdown 2023-09-30T00:48:02.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:02.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 1 -- 172.21.15.40:0/3326160130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee3c13ac20 msgr2=0x7fee3c151120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 1 --2- 172.21.15.40:0/3326160130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee3c13ac20 0x7fee3c151120 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fee3400b6c0 tx=0x7fee34017000 comp rx=0 tx=0).stop 2023-09-30T00:48:02.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 1 -- 172.21.15.40:0/3326160130 shutdown_connections 2023-09-30T00:48:02.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 1 --2- 172.21.15.40:0/3326160130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee3c13ac20 0x7fee3c151120 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 1 --2- 172.21.15.40:0/3326160130 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fee3c0a5fb0 0x7fee3c0aa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 1 --2- 172.21.15.40:0/3326160130 >> v2:172.21.15.135:6800/116426 conn(0x7fee20068890 0x7fee2006ad40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 1 --2- 172.21.15.40:0/3326160130 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fee3c0b1460 0x7fee3c13e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 1 -- 172.21.15.40:0/3326160130 >> 172.21.15.40:0/3326160130 conn(0x7fee3c0631f0 msgr2=0x7fee3c144970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:02.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 1 -- 172.21.15.40:0/3326160130 shutdown_connections 2023-09-30T00:48:02.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 1 -- 172.21.15.40:0/3326160130 wait complete. 2023-09-30T00:48:02.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.328+0000 7fee4371a700 1 librados: shutdown 2023-09-30T00:48:02.343 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.5 2023-09-30T00:48:02.343 DEBUG:teuthology.parallel:result is None 2023-09-30T00:48:02.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.552+0000 7fdbd9335700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:48:02.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.552+0000 7fdbd9335700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:48:02.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.553+0000 7fdbd9335700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:02.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.553+0000 7fdbd9335700 1 -- 172.21.15.40:0/1844732692 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdbd40ab180 con 0x7fdbd413e1d0 2023-09-30T00:48:02.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.557+0000 7fdbbaffd700 1 -- 172.21.15.40:0/1844732692 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdbd40ab180 con 0x7fdbd413e1d0 2023-09-30T00:48:02.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.557+0000 7fdbbaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:48:02.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.557+0000 7fdbbaffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:02.558 INFO:teuthology.orchestra.run.smithi040.stdout:81604378630 2023-09-30T00:48:02.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 10 librados: watch_flush enter 2023-09-30T00:48:02.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 10 librados: watch_flush exit 2023-09-30T00:48:02.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 20 client.4359.objecter shutdown clearing up homeless session... 2023-09-30T00:48:02.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 10 client.4359.objecter successfully canceled tick 2023-09-30T00:48:02.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 1 -- 172.21.15.40:0/1844732692 >> v2:172.21.15.135:6800/116426 conn(0x7fdbc0068910 msgr2=0x7fdbc006adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 1 --2- 172.21.15.40:0/1844732692 >> v2:172.21.15.135:6800/116426 conn(0x7fdbc0068910 0x7fdbc006adc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40016a0 tx=0x7fdbc4007b30 comp rx=0 tx=0).stop 2023-09-30T00:48:02.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 10 monclient: shutdown 2023-09-30T00:48:02.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:02.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 1 -- 172.21.15.40:0/1844732692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd413e1d0 msgr2=0x7fdbd40abb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 1 --2- 172.21.15.40:0/1844732692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd413e1d0 0x7fdbd40abb80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdbc800c790 tx=0x7fdbc8007f40 comp rx=0 tx=0).stop 2023-09-30T00:48:02.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 1 -- 172.21.15.40:0/1844732692 shutdown_connections 2023-09-30T00:48:02.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 1 --2- 172.21.15.40:0/1844732692 >> v2:172.21.15.135:6800/116426 conn(0x7fdbc0068910 0x7fdbc006adc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 1 --2- 172.21.15.40:0/1844732692 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdbd4146d00 0x7fdbd40ae4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 1 --2- 172.21.15.40:0/1844732692 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdbd40a5710 0x7fdbd40a98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 1 --2- 172.21.15.40:0/1844732692 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdbd413e1d0 0x7fdbd40abb80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 1 -- 172.21.15.40:0/1844732692 >> 172.21.15.40:0/1844732692 conn(0x7fdbd4069080 msgr2=0x7fdbd413a630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:02.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 1 -- 172.21.15.40:0/1844732692 shutdown_connections 2023-09-30T00:48:02.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 1 -- 172.21.15.40:0/1844732692 wait complete. 2023-09-30T00:48:02.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.560+0000 7fdbd9335700 1 librados: shutdown 2023-09-30T00:48:02.575 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378630 got 81604378630 for osd.1 2023-09-30T00:48:02.575 DEBUG:teuthology.parallel:result is None 2023-09-30T00:48:02.575 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T00:48:02.575 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:48:02.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.714+0000 7fb540d16700 10 monclient: get_monmap_and_config 2023-09-30T00:48:02.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.714+0000 7fb540d16700 10 monclient: build_initial_monmap 2023-09-30T00:48:02.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.714+0000 7fb540d16700 10 monclient: monmap: 2023-09-30T00:48:02.715 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:02.715 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:02.715 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:02.715089+0000 2023-09-30T00:48:02.715 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:02.715089+0000 2023-09-30T00:48:02.715 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:02.715 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:02.716 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:02.716 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:02.716 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:02.716 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:02.716 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:02.716 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:02.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.714+0000 7fb540d16700 1 Processor -- start 2023-09-30T00:48:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.714+0000 7fb540d16700 1 -- start start 2023-09-30T00:48:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.714+0000 7fb540d16700 10 monclient: init 2023-09-30T00:48:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.715+0000 7fb540d16700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.715+0000 7fb540d16700 10 monclient: _add_conns ranks=[4,3,1,0,2] 2023-09-30T00:48:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.715+0000 7fb540d16700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb53c0b1460 0x7fb53c0b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.715+0000 7fb540d16700 10 monclient(hunting): picked mon.noname-c con 0x7fb53c0b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.715+0000 7fb540d16700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb53c0a5fb0 0x7fb53c141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.715+0000 7fb540d16700 10 monclient(hunting): picked mon.noname-a con 0x7fb53c0a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:02.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.715+0000 7fb540d16700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb53c0af410 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.715+0000 7fb540d16700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53c0a71f0 con 0x7fb53c0af410 2023-09-30T00:48:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.715+0000 7fb540d16700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.715+0000 7fb540d16700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb53c0a74d0 con 0x7fb53c0a5fb0 2023-09-30T00:48:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.715+0000 7fb540d16700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.715+0000 7fb540d16700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb53c0a63c0 con 0x7fb53c0b1460 2023-09-30T00:48:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.715+0000 7fb540d16700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.715+0000 7fb540d16700 10 monclient(hunting): authenticate will time out at 5344.016709s 2023-09-30T00:48:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.716+0000 7fb53a59c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb53c0b1460 0x7fb53c0b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.717+0000 7fb53ad9d700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fb53c0af410 0x7fb53c0af870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:58128/0 (socket says 172.21.15.40:58128) 2023-09-30T00:48:02.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.717+0000 7fb53ad9d700 1 -- 172.21.15.40:0/4072340514 learned_addr learned my addr 172.21.15.40:0/4072340514 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:48:02.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.717+0000 7fb53a59c700 10 monclient(hunting): get_auth_request con 0x7fb53c0b1460 auth_method 0 2023-09-30T00:48:02.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.717+0000 7fb53a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:02.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.717+0000 7fb53a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:02.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.717+0000 7fb53a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:02.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.718+0000 7fb53a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:02.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.718+0000 7fb53a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:02.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.718+0000 7fb53a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:02.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.720+0000 7fb53a59c700 10 monclient(hunting): handle_auth_done global_id 4384 payload 274 2023-09-30T00:48:02.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.720+0000 7fb53a59c700 1 -- 172.21.15.40:0/4072340514 >> v2:172.21.15.40:3300/0 conn(0x7fb53c0a5fb0 msgr2=0x7fb53c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:02.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.720+0000 7fb53a59c700 1 --2- 172.21.15.40:0/4072340514 >> v2:172.21.15.40:3300/0 conn(0x7fb53c0a5fb0 0x7fb53c141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.720+0000 7fb53a59c700 1 -- 172.21.15.40:0/4072340514 >> v1:172.21.15.135:6789/0 conn(0x7fb53c0af410 legacy=0x7fb53c0af870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.720+0000 7fb53a59c700 10 monclient: _finish_hunting 0 2023-09-30T00:48:02.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.720+0000 7fb53a59c700 1 monclient: found mon.noname-c 2023-09-30T00:48:02.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.720+0000 7fb53a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:02.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.720+0000 7fb53a59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:48:02.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.720+0000 7fb53a59c700 1 -- 172.21.15.40:0/4072340514 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53c0a56d0 con 0x7fb53c0b1460 2023-09-30T00:48:02.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.723+0000 7fb53a59c700 1 --2- 172.21.15.40:0/4072340514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb53c0b1460 0x7fb53c0b1830 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb534009e60 tx=0x7fb534009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5eb86c37c0ebcfd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb53959a700 1 -- 172.21.15.40:0/4072340514 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53400e070 con 0x7fb53c0b1460 2023-09-30T00:48:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb53959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb53959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb53959a700 10 monclient: dump: 2023-09-30T00:48:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:02.731 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb53959a700 10 monclient: _finish_auth 0 2023-09-30T00:48:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb53959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb53959a700 1 -- 172.21.15.40:0/4072340514 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53400bbf0 con 0x7fb53c0b1460 2023-09-30T00:48:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb53959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb53959a700 1 -- 172.21.15.40:0/4072340514 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb534012430 con 0x7fb53c0b1460 2023-09-30T00:48:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb540d16700 5 monclient: authenticate success, global_id 4384 2023-09-30T00:48:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb540d16700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb53959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb540d16700 4 set_mon_vals no callback set 2023-09-30T00:48:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb53959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.729+0000 7fb53959a700 10 monclient: dump: 2023-09-30T00:48:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:02.733 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.730+0000 7fb540d16700 10 monclient: shutdown 2023-09-30T00:48:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.730+0000 7fb540d16700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.730+0000 7fb540d16700 1 -- 172.21.15.40:0/4072340514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb53c0b1460 msgr2=0x7fb53c0b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.730+0000 7fb540d16700 1 --2- 172.21.15.40:0/4072340514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb53c0b1460 0x7fb53c0b1830 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb534009e60 tx=0x7fb534009780 comp rx=0 tx=0).stop 2023-09-30T00:48:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.730+0000 7fb540d16700 1 -- 172.21.15.40:0/4072340514 shutdown_connections 2023-09-30T00:48:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.730+0000 7fb540d16700 1 --2- 172.21.15.40:0/4072340514 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb53c0b1460 0x7fb53c0b1830 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.730+0000 7fb540d16700 1 --2- 172.21.15.40:0/4072340514 >> v2:172.21.15.40:3300/0 conn(0x7fb53c0a5fb0 0x7fb53c141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.730+0000 7fb540d16700 1 -- 172.21.15.40:0/4072340514 >> 172.21.15.40:0/4072340514 conn(0x7fb53c0631f0 msgr2=0x7fb53c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.730+0000 7fb540d16700 1 -- 172.21.15.40:0/4072340514 shutdown_connections 2023-09-30T00:48:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.730+0000 7fb540d16700 1 -- 172.21.15.40:0/4072340514 wait complete. 2023-09-30T00:48:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.730+0000 7fb540d16700 10 monclient: build_initial_monmap 2023-09-30T00:48:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.731+0000 7fb540d16700 10 monclient: monmap: 2023-09-30T00:48:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:02.737 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.731+0000 7fb540d16700 1 librados: starting msgr at 2023-09-30T00:48:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.731+0000 7fb540d16700 1 librados: starting objecter 2023-09-30T00:48:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.731+0000 7fb540d16700 1 Processor -- start 2023-09-30T00:48:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.731+0000 7fb540d16700 1 -- start start 2023-09-30T00:48:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.731+0000 7fb540d16700 1 librados: setting wanted keys 2023-09-30T00:48:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.731+0000 7fb540d16700 1 librados: calling monclient init 2023-09-30T00:48:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.731+0000 7fb540d16700 10 monclient: init 2023-09-30T00:48:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:48:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb53c0a5fb0 0x7fb53c0a85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 10 monclient(hunting): picked mon.noname-b con 0x7fb53c0a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb53c0af410 0x7fb53c0a9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 10 monclient(hunting): picked mon.noname-c con 0x7fb53c0af410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c0b1460 0x7fb53c13d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 10 monclient(hunting): picked mon.noname-a con 0x7fb53c0b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c0aaaa0 con 0x7fb53c0b1460 2023-09-30T00:48:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c0a9130 con 0x7fb53c0a5fb0 2023-09-30T00:48:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c0ae600 con 0x7fb53c0af410 2023-09-30T00:48:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.732+0000 7fb540d16700 10 monclient(hunting): authenticate will time out at 5344.033362s 2023-09-30T00:48:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.733+0000 7fb53a59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb53c0a5fb0 0x7fb53c0a85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.733+0000 7fb53a59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb53c0a5fb0 0x7fb53c0a85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:41480/0 (socket says 172.21.15.40:41480) 2023-09-30T00:48:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.733+0000 7fb53a59c700 1 -- 172.21.15.40:0/2413803076 learned_addr learned my addr 172.21.15.40:0/2413803076 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.733+0000 7fb53a59c700 10 monclient(hunting): get_auth_request con 0x7fb53c0a5fb0 auth_method 0 2023-09-30T00:48:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.733+0000 7fb53a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.733+0000 7fb53a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.733+0000 7fb53a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.734+0000 7fb53a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.734+0000 7fb53a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.734+0000 7fb53a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.737+0000 7fb53a59c700 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2023-09-30T00:48:02.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.737+0000 7fb53a59c700 1 -- 172.21.15.40:0/2413803076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb53c0af410 msgr2=0x7fb53c0a9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.737+0000 7fb53a59c700 1 --2- 172.21.15.40:0/2413803076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb53c0af410 0x7fb53c0a9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.737+0000 7fb53a59c700 1 -- 172.21.15.40:0/2413803076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c0b1460 msgr2=0x7fb53c13d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.737+0000 7fb53a59c700 1 --2- 172.21.15.40:0/2413803076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c0b1460 0x7fb53c13d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.737+0000 7fb53a59c700 10 monclient: _finish_hunting 0 2023-09-30T00:48:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.737+0000 7fb53a59c700 1 monclient: found mon.noname-b 2023-09-30T00:48:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.737+0000 7fb53a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.737+0000 7fb53a59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.737+0000 7fb53a59c700 1 -- 172.21.15.40:0/2413803076 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb534009a20 con 0x7fb53c0a5fb0 2023-09-30T00:48:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.740+0000 7fb53a59c700 1 --2- 172.21.15.40:0/2413803076 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb53c0a5fb0 0x7fb53c0a85f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb53c0a6500 tx=0x7fb534019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.740+0000 7fb5327fc700 10 client.?.objecter ms_handle_connect 0x7fb53c0a5fb0 2023-09-30T00:48:02.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.740+0000 7fb5327fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb5327fc700 1 -- 172.21.15.40:0/2413803076 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb534027470 con 0x7fb53c0a5fb0 2023-09-30T00:48:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb5327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb5327fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb5327fc700 10 monclient: dump: 2023-09-30T00:48:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:02.747 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb5327fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb5327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb5327fc700 1 -- 172.21.15.40:0/2413803076 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5340275d0 con 0x7fb53c0a5fb0 2023-09-30T00:48:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb5327fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb5327fc700 1 -- 172.21.15.40:0/2413803076 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb534027990 con 0x7fb53c0a5fb0 2023-09-30T00:48:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb5327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb5327fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb5327fc700 10 monclient: dump: 2023-09-30T00:48:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:02.748 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:02.749 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb540d16700 5 monclient: authenticate success, global_id 4387 2023-09-30T00:48:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb540d16700 10 monclient: _renew_subs 2023-09-30T00:48:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb538d99700 4 set_mon_vals no callback set 2023-09-30T00:48:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb540d16700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.744+0000 7fb540d16700 1 -- 172.21.15.40:0/2413803076 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53c13f490 con 0x7fb53c0a5fb0 2023-09-30T00:48:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.745+0000 7fb540d16700 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.745+0000 7fb540d16700 10 monclient: _renew_subs 2023-09-30T00:48:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.745+0000 7fb540d16700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.745+0000 7fb540d16700 1 -- 172.21.15.40:0/2413803076 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53c0a8e90 con 0x7fb53c0a5fb0 2023-09-30T00:48:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.745+0000 7fb540d16700 1 librados: init done 2023-09-30T00:48:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.746+0000 7fb51bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.746+0000 7fb51bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.746+0000 7fb51bfff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:02.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:02.746+0000 7fb51bfff700 1 -- 172.21.15.40:0/2413803076 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb53c0a9350 con 0x7fb53c0a5fb0 2023-09-30T00:48:03.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.060+0000 7fb5327fc700 1 -- 172.21.15.40:0/2413803076 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fb534024070 con 0x7fb53c0a5fb0 2023-09-30T00:48:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.060+0000 7fb5327fc700 1 --2- 172.21.15.40:0/2413803076 >> v2:172.21.15.135:6800/116426 conn(0x7fb51c068910 0x7fb51c06adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.060+0000 7fb5327fc700 1 -- 172.21.15.40:0/2413803076 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fb534086d60 con 0x7fb53c0a5fb0 2023-09-30T00:48:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.060+0000 7fb5327fc700 10 client.4387.objecter ms_dispatch 0x7fb53c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.060+0000 7fb5327fc700 3 client.4387.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.060+0000 7fb5327fc700 3 client.4387.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.061+0000 7fb5327fc700 20 client.4387.objecter dump_active .. 0 homeless 2023-09-30T00:48:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.061+0000 7fb539d9b700 1 --2- 172.21.15.40:0/2413803076 >> v2:172.21.15.135:6800/116426 conn(0x7fb51c068910 0x7fb51c06adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.061+0000 7fb539d9b700 10 monclient: get_auth_request con 0x7fb51c068910 auth_method 0 2023-09-30T00:48:03.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.061+0000 7fb539d9b700 1 --2- 172.21.15.40:0/2413803076 >> v2:172.21.15.135:6800/116426 conn(0x7fb51c068910 0x7fb51c06adc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb524006fd0 tx=0x7fb524006db0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:03.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.061+0000 7fb5327fc700 10 client.4387.objecter ms_handle_connect 0x7fb51c068910 2023-09-30T00:48:03.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.070+0000 7fb5327fc700 1 -- 172.21.15.40:0/2413803076 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb534054600 con 0x7fb53c0a5fb0 2023-09-30T00:48:03.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.070+0000 7fb5327fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:03.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.070+0000 7fb5327fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:03.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.293+0000 7fb51bfff700 1 -- 172.21.15.40:0/2413803076 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb53c064a00 con 0x7fb51c068910 2023-09-30T00:48:03.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.296+0000 7fb5327fc700 1 -- 172.21.15.40:0/2413803076 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48056 (secure 0 0 0) 0x7fb53c064a00 con 0x7fb51c068910 2023-09-30T00:48:03.297 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:03.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 10 librados: watch_flush enter 2023-09-30T00:48:03.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 10 librados: watch_flush exit 2023-09-30T00:48:03.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 20 client.4387.objecter shutdown clearing up homeless session... 2023-09-30T00:48:03.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 10 client.4387.objecter successfully canceled tick 2023-09-30T00:48:03.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 1 -- 172.21.15.40:0/2413803076 >> v2:172.21.15.135:6800/116426 conn(0x7fb51c068910 msgr2=0x7fb51c06adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:03.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 1 --2- 172.21.15.40:0/2413803076 >> v2:172.21.15.135:6800/116426 conn(0x7fb51c068910 0x7fb51c06adc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb524006fd0 tx=0x7fb524006db0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 10 monclient: shutdown 2023-09-30T00:48:03.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:03.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 1 -- 172.21.15.40:0/2413803076 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb53c0a5fb0 msgr2=0x7fb53c0a85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:03.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 1 --2- 172.21.15.40:0/2413803076 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb53c0a5fb0 0x7fb53c0a85f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb53c0a6500 tx=0x7fb534019000 comp rx=0 tx=0).stop 2023-09-30T00:48:03.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 1 -- 172.21.15.40:0/2413803076 shutdown_connections 2023-09-30T00:48:03.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 1 --2- 172.21.15.40:0/2413803076 >> v2:172.21.15.135:6800/116426 conn(0x7fb51c068910 0x7fb51c06adc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.303 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 1 --2- 172.21.15.40:0/2413803076 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb53c0a5fb0 0x7fb53c0a85f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.303 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 1 --2- 172.21.15.40:0/2413803076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb53c0b1460 0x7fb53c13d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.303 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 1 --2- 172.21.15.40:0/2413803076 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb53c0af410 0x7fb53c0a9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.303 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 1 -- 172.21.15.40:0/2413803076 >> 172.21.15.40:0/2413803076 conn(0x7fb53c0631f0 msgr2=0x7fb53c0b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:03.303 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 1 -- 172.21.15.40:0/2413803076 shutdown_connections 2023-09-30T00:48:03.303 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.298+0000 7fb51bfff700 1 -- 172.21.15.40:0/2413803076 wait complete. 2023-09-30T00:48:03.304 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.299+0000 7fb51bfff700 1 librados: shutdown 2023-09-30T00:48:03.304 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-09-30T00:48:03.313 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2023-09-30T00:48:01.873509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43464,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931256,"statfs":{"total":773094113280,"available":773049606144,"internally_reserved":0,"allocated":2506752,"data_stored":1383352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":18,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001342"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.785504+0000","last_change":"2023-09-30T00:47:54.691712+0000","last_active":"2023-09-30T00:47:54.785504+0000","last_peered":"2023-09-30T00:47:54.785504+0000","last_clean":"2023-09-30T00:47:54.785504+0000","last_became_active":"2023-09-30T00:47:54.672127+0000","last_became_peered":"2023-09-30T00:47:54.672127+0000","last_unstale":"2023-09-30T00:47:54.785504+0000","last_undegraded":"2023-09-30T00:47:54.785504+0000","last_fullsized":"2023-09-30T00:47:54.785504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:32.704584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.160955+0000","last_change":"2023-09-30T00:47:54.749019+0000","last_active":"2023-09-30T00:47:55.160955+0000","last_peered":"2023-09-30T00:47:55.160955+0000","last_clean":"2023-09-30T00:47:55.160955+0000","last_became_active":"2023-09-30T00:47:54.702773+0000","last_became_peered":"2023-09-30T00:47:54.702773+0000","last_unstale":"2023-09-30T00:47:55.160955+0000","last_undegraded":"2023-09-30T00:47:55.160955+0000","last_fullsized":"2023-09-30T00:47:55.160955+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:07:10.081341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.809132+0000","last_change":"2023-09-30T00:47:54.157108+0000","last_active":"2023-09-30T00:47:54.809132+0000","last_peered":"2023-09-30T00:47:54.809132+0000","last_clean":"2023-09-30T00:47:54.809132+0000","last_became_active":"2023-09-30T00:47:54.139464+0000","last_became_peered":"2023-09-30T00:47:54.139464+0000","last_unstale":"2023-09-30T00:47:54.809132+0000","last_undegraded":"2023-09-30T00:47:54.809132+0000","last_fullsized":"2023-09-30T00:47:54.809132+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:42:38.117935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.795839+0000","last_change":"2023-09-30T00:47:53.993668+0000","last_active":"2023-09-30T00:47:54.795839+0000","last_peered":"2023-09-30T00:47:54.795839+0000","last_clean":"2023-09-30T00:47:54.795839+0000","last_became_active":"2023-09-30T00:47:53.951882+0000","last_became_peered":"2023-09-30T00:47:53.951882+0000","last_unstale":"2023-09-30T00:47:54.795839+0000","last_undegraded":"2023-09-30T00:47:54.795839+0000","last_fullsized":"2023-09-30T00:47:54.795839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:15:02.278001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'77","reported_seq":114,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:47:57.719324+0000","last_change":"2023-09-30T00:47:47.347565+0000","last_active":"2023-09-30T00:47:57.719324+0000","last_peered":"2023-09-30T00:47:57.719324+0000","last_clean":"2023-09-30T00:47:57.719324+0000","last_became_active":"2023-09-30T00:47:46.406306+0000","last_became_peered":"2023-09-30T00:47:46.406306+0000","last_unstale":"2023-09-30T00:47:57.719324+0000","last_undegraded":"2023-09-30T00:47:57.719324+0000","last_fullsized":"2023-09-30T00:47:57.719324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:40.867840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:40.867840+0000","last_clean_scrub_stamp":"2023-09-30T00:47:40.867840+0000","objects_scrubbed":0,"log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:17:21.219288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.780151+0000","last_change":"2023-09-30T00:47:54.729272+0000","last_active":"2023-09-30T00:47:54.780151+0000","last_peered":"2023-09-30T00:47:54.780151+0000","last_clean":"2023-09-30T00:47:54.780151+0000","last_became_active":"2023-09-30T00:47:54.710752+0000","last_became_peered":"2023-09-30T00:47:54.710752+0000","last_unstale":"2023-09-30T00:47:54.780151+0000","last_undegraded":"2023-09-30T00:47:54.780151+0000","last_fullsized":"2023-09-30T00:47:54.780151+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:59:21.956005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.239361+0000","last_change":"2023-09-30T00:47:53.773125+0000","last_active":"2023-09-30T00:47:55.239361+0000","last_peered":"2023-09-30T00:47:55.239361+0000","last_clean":"2023-09-30T00:47:55.239361+0000","last_became_active":"2023-09-30T00:47:53.761460+0000","last_became_peered":"2023-09-30T00:47:53.761460+0000","last_unstale":"2023-09-30T00:47:55.239361+0000","last_undegraded":"2023-09-30T00:47:55.239361+0000","last_fullsized":"2023-09-30T00:47:55.239361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:55.538707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.791094+0000","last_change":"2023-09-30T00:47:54.294197+0000","last_active":"2023-09-30T00:47:54.791094+0000","last_peered":"2023-09-30T00:47:54.791094+0000","last_clean":"2023-09-30T00:47:54.791094+0000","last_became_active":"2023-09-30T00:47:54.279706+0000","last_became_peered":"2023-09-30T00:47:54.279706+0000","last_unstale":"2023-09-30T00:47:54.791094+0000","last_undegraded":"2023-09-30T00:47:54.791094+0000","last_fullsized":"2023-09-30T00:47:54.791094+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:24:53.339591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.240181+0000","last_change":"2023-09-30T00:47:54.352713+0000","last_active":"2023-09-30T00:47:55.240181+0000","last_peered":"2023-09-30T00:47:55.240181+0000","last_clean":"2023-09-30T00:47:55.240181+0000","last_became_active":"2023-09-30T00:47:54.305507+0000","last_became_peered":"2023-09-30T00:47:54.305507+0000","last_unstale":"2023-09-30T00:47:55.240181+0000","last_undegraded":"2023-09-30T00:47:55.240181+0000","last_fullsized":"2023-09-30T00:47:55.240181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:21:08.873800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345928,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2550000000000008}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.238}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.448}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.033999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.835000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.471}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.646999999999998}]}]},{"osd":2,"up_from":20,"seq":85899345927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.42}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.690999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.763999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.837}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.577999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.646000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.442}]}]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":614400,"data_stored":471299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.396000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5039999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.419}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.076000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8010000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.814}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.652999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.098000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.222000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.800999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.609000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.375}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.542999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.617999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.0739999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3190000000000008}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.356000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.914999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.492000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.476000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.484999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6199999999999992}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.605}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.693999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.975}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.768999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.376999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.343}]}]},{"osd":0,"up_from":20,"seq":85899345928,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.586}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.856}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.518000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.780999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.098000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.289999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.065999999999999}]}]},{"osd":1,"up_from":19,"seq":81604378631,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6044,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365796,"statfs":{"total":96636764160,"available":96630575104,"internally_reserved":0,"allocated":614400,"data_stored":471299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.711}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.259}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.108000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.56}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.704000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.382999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.912000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:48:03.315 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:48:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.453+0000 7f0b30acf700 10 monclient: get_monmap_and_config 2023-09-30T00:48:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.453+0000 7f0b30acf700 10 monclient: build_initial_monmap 2023-09-30T00:48:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.453+0000 7f0b30acf700 10 monclient: monmap: 2023-09-30T00:48:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:03.454413+0000 2023-09-30T00:48:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:03.454413+0000 2023-09-30T00:48:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:03.455 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:03.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 1 Processor -- start 2023-09-30T00:48:03.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 1 -- start start 2023-09-30T00:48:03.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 10 monclient: init 2023-09-30T00:48:03.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:03.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-09-30T00:48:03.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b2c0b1460 0x7f0b2c0b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:03.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 10 monclient(hunting): picked mon.noname-b con 0x7f0b2c0b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:03.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0b2c0a5fb0 0x7f0b2c141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:03.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 10 monclient(hunting): picked mon.noname-a con 0x7f0b2c0a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:03.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0b2c0af410 0x7f0b2c0af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:03.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 10 monclient(hunting): picked mon.noname-c con 0x7f0b2c0af410 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:03.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:03.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b2c0a71f0 con 0x7f0b2c0a5fb0 2023-09-30T00:48:03.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:03.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b2c0a74d0 con 0x7f0b2c0af410 2023-09-30T00:48:03.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:03.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c0a63c0 con 0x7f0b2c0b1460 2023-09-30T00:48:03.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:03.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.454+0000 7f0b30acf700 10 monclient(hunting): authenticate will time out at 5344.755837s 2023-09-30T00:48:03.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.455+0000 7f0b2ad9d700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0b2c0af410 0x7f0b2c0af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:03.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.455+0000 7f0b21d9b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0b2c0a5fb0 0x7f0b2c141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:03.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.455+0000 7f0b2a59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b2c0b1460 0x7f0b2c0b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:03.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.455+0000 7f0b2ad9d700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0b2c0af410 0x7f0b2c0af890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:41496/0 (socket says 172.21.15.40:41496) 2023-09-30T00:48:03.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.455+0000 7f0b2ad9d700 1 -- 172.21.15.40:0/655595587 learned_addr learned my addr 172.21.15.40:0/655595587 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:03.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.455+0000 7f0b2ad9d700 10 monclient(hunting): get_auth_request con 0x7f0b2c0af410 auth_method 0 2023-09-30T00:48:03.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.455+0000 7f0b2ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:03.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.455+0000 7f0b2ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:03.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.455+0000 7f0b2ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:03.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.455+0000 7f0b21d9b700 10 monclient(hunting): get_auth_request con 0x7f0b2c0a5fb0 auth_method 0 2023-09-30T00:48:03.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.455+0000 7f0b21d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:03.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.455+0000 7f0b21d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:03.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.455+0000 7f0b21d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:03.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.456+0000 7f0b2a59c700 10 monclient(hunting): get_auth_request con 0x7f0b2c0b1460 auth_method 0 2023-09-30T00:48:03.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.456+0000 7f0b2a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:03.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.456+0000 7f0b2a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:03.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.456+0000 7f0b2a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:03.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.456+0000 7f0b2ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:03.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.456+0000 7f0b2ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:03.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.456+0000 7f0b2ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:03.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.456+0000 7f0b21d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:03.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.456+0000 7f0b21d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:03.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.456+0000 7f0b21d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:03.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.456+0000 7f0b2a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:03.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.456+0000 7f0b2a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:03.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.456+0000 7f0b2a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:03.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.458+0000 7f0b2ad9d700 10 monclient(hunting): handle_auth_done global_id 4390 payload 274 2023-09-30T00:48:03.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.458+0000 7f0b2ad9d700 1 -- 172.21.15.40:0/655595587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b2c0b1460 msgr2=0x7f0b2c0b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:03.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.458+0000 7f0b2ad9d700 1 --2- 172.21.15.40:0/655595587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b2c0b1460 0x7f0b2c0b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.458+0000 7f0b2ad9d700 1 -- 172.21.15.40:0/655595587 >> v2:172.21.15.40:3300/0 conn(0x7f0b2c0a5fb0 msgr2=0x7f0b2c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:03.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.458+0000 7f0b2ad9d700 1 --2- 172.21.15.40:0/655595587 >> v2:172.21.15.40:3300/0 conn(0x7f0b2c0a5fb0 0x7f0b2c141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.458+0000 7f0b2ad9d700 10 monclient: _finish_hunting 0 2023-09-30T00:48:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.458+0000 7f0b2ad9d700 1 monclient: found mon.noname-c 2023-09-30T00:48:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.458+0000 7f0b2ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.458+0000 7f0b2ad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:48:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.458+0000 7f0b2ad9d700 1 -- 172.21.15.40:0/655595587 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c0a56d0 con 0x7f0b2c0af410 2023-09-30T00:48:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.460+0000 7f0b2ad9d700 1 --2- 172.21.15.40:0/655595587 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b2c0af410 0x7f0b2c0af890 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c0b05f0 tx=0x7f0b24009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=335973bc9e15a1f5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:03.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b29d9b700 1 -- 172.21.15.40:0/655595587 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2400e070 con 0x7f0b2c0af410 2023-09-30T00:48:03.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b29d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:03.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b29d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:03.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b29d9b700 10 monclient: dump: 2023-09-30T00:48:03.465 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:03.465 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:03.465 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:03.465 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:03.465 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:03.465 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:03.465 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:03.465 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:03.465 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:03.465 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:03.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b29d9b700 10 monclient: _finish_auth 0 2023-09-30T00:48:03.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b29d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:03.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b29d9b700 1 -- 172.21.15.40:0/655595587 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2400b970 con 0x7f0b2c0af410 2023-09-30T00:48:03.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b29d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:03.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b29d9b700 1 -- 172.21.15.40:0/655595587 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b240092e0 con 0x7f0b2c0af410 2023-09-30T00:48:03.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b30acf700 5 monclient: authenticate success, global_id 4390 2023-09-30T00:48:03.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b30acf700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:03.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b30acf700 4 set_mon_vals no callback set 2023-09-30T00:48:03.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b29d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:03.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b29d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:03.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b29d9b700 10 monclient: dump: 2023-09-30T00:48:03.466 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:03.467 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:03.467 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:03.467 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:03.467 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:03.467 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:03.467 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:03.467 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:03.467 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:03.467 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b30acf700 10 monclient: shutdown 2023-09-30T00:48:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b30acf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b30acf700 1 -- 172.21.15.40:0/655595587 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b2c0af410 msgr2=0x7f0b2c0af890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b30acf700 1 --2- 172.21.15.40:0/655595587 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b2c0af410 0x7f0b2c0af890 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c0b05f0 tx=0x7f0b24009840 comp rx=0 tx=0).stop 2023-09-30T00:48:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b30acf700 1 -- 172.21.15.40:0/655595587 shutdown_connections 2023-09-30T00:48:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b30acf700 1 --2- 172.21.15.40:0/655595587 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b2c0af410 0x7f0b2c0af890 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.464+0000 7f0b30acf700 1 --2- 172.21.15.40:0/655595587 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b2c0b1460 0x7f0b2c0b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.465+0000 7f0b30acf700 1 --2- 172.21.15.40:0/655595587 >> v2:172.21.15.40:3300/0 conn(0x7f0b2c0a5fb0 0x7f0b2c141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.465+0000 7f0b30acf700 1 -- 172.21.15.40:0/655595587 >> 172.21.15.40:0/655595587 conn(0x7f0b2c0631f0 msgr2=0x7f0b2c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.465+0000 7f0b30acf700 1 -- 172.21.15.40:0/655595587 shutdown_connections 2023-09-30T00:48:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.465+0000 7f0b30acf700 1 -- 172.21.15.40:0/655595587 wait complete. 2023-09-30T00:48:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.465+0000 7f0b30acf700 10 monclient: build_initial_monmap 2023-09-30T00:48:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.465+0000 7f0b30acf700 10 monclient: monmap: 2023-09-30T00:48:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:03.472 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.466+0000 7f0b30acf700 1 librados: starting msgr at 2023-09-30T00:48:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.466+0000 7f0b30acf700 1 librados: starting objecter 2023-09-30T00:48:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.466+0000 7f0b30acf700 1 Processor -- start 2023-09-30T00:48:03.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.466+0000 7f0b30acf700 1 -- start start 2023-09-30T00:48:03.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.466+0000 7f0b30acf700 1 librados: setting wanted keys 2023-09-30T00:48:03.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.466+0000 7f0b30acf700 1 librados: calling monclient init 2023-09-30T00:48:03.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.466+0000 7f0b30acf700 10 monclient: init 2023-09-30T00:48:03.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.466+0000 7f0b30acf700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:03.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.466+0000 7f0b30acf700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:48:03.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b30acf700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b2c0a5fb0 0x7f0b2c0a7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:03.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b30acf700 10 monclient(hunting): picked mon.noname-c con 0x7f0b2c0a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:03.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b30acf700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b2c0af410 0x7f0b2c13cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:03.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b30acf700 10 monclient(hunting): picked mon.noname-b con 0x7f0b2c0af410 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:03.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b30acf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b2c0b1460 0x7f0b2c13d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:03.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b30acf700 10 monclient(hunting): picked mon.noname-a con 0x7f0b2c0b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:03.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b30acf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:03.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b30acf700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c13e850 con 0x7f0b2c0b1460 2023-09-30T00:48:03.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b30acf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:03.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b30acf700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c1456d0 con 0x7f0b2c0af410 2023-09-30T00:48:03.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b30acf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:03.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b30acf700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c13ea80 con 0x7f0b2c0a5fb0 2023-09-30T00:48:03.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b30acf700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:03.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b30acf700 10 monclient(hunting): authenticate will time out at 5344.768072s 2023-09-30T00:48:03.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b2ad9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b2c0b1460 0x7f0b2c13d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:03.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b2a59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b2c0a5fb0 0x7f0b2c0a7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:03.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b21d9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b2c0af410 0x7f0b2c13cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:03.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.467+0000 7f0b2ad9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b2c0b1460 0x7f0b2c13d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52442/0 (socket says 172.21.15.40:52442) 2023-09-30T00:48:03.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2ad9d700 1 -- 172.21.15.40:0/2076080530 learned_addr learned my addr 172.21.15.40:0/2076080530 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:03.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2ad9d700 10 monclient(hunting): get_auth_request con 0x7f0b2c0b1460 auth_method 0 2023-09-30T00:48:03.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:03.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:03.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:03.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2a59c700 10 monclient(hunting): get_auth_request con 0x7f0b2c0a5fb0 auth_method 0 2023-09-30T00:48:03.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:03.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:03.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:03.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b21d9b700 10 monclient(hunting): get_auth_request con 0x7f0b2c0af410 auth_method 0 2023-09-30T00:48:03.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b21d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:03.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b21d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:03.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b21d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:03.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:03.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:03.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:03.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b21d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:03.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b21d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:03.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b21d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:03.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:03.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:03.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.468+0000 7f0b2a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:03.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.471+0000 7f0b21d9b700 10 monclient(hunting): handle_auth_done global_id 4393 payload 824 2023-09-30T00:48:03.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.471+0000 7f0b21d9b700 1 -- 172.21.15.40:0/2076080530 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b2c0a5fb0 msgr2=0x7f0b2c0a7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:03.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.471+0000 7f0b21d9b700 1 --2- 172.21.15.40:0/2076080530 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b2c0a5fb0 0x7f0b2c0a7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.471+0000 7f0b21d9b700 1 -- 172.21.15.40:0/2076080530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b2c0b1460 msgr2=0x7f0b2c13d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:03.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.471+0000 7f0b21d9b700 1 --2- 172.21.15.40:0/2076080530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b2c0b1460 0x7f0b2c13d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.471+0000 7f0b21d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:48:03.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.471+0000 7f0b21d9b700 1 monclient: found mon.noname-b 2023-09-30T00:48:03.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.472+0000 7f0b21d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:03.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.472+0000 7f0b21d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:03.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.472+0000 7f0b21d9b700 1 -- 172.21.15.40:0/2076080530 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b24009ae0 con 0x7f0b2c0af410 2023-09-30T00:48:03.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.474+0000 7f0b21d9b700 1 --2- 172.21.15.40:0/2076080530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b2c0af410 0x7f0b2c13cc30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400ea80 tx=0x7f0b140091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:03.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.474+0000 7f0b22ffd700 10 client.?.objecter ms_handle_connect 0x7f0b2c0af410 2023-09-30T00:48:03.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.474+0000 7f0b22ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:03.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.477+0000 7f0b22ffd700 1 -- 172.21.15.40:0/2076080530 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1400eec0 con 0x7f0b2c0af410 2023-09-30T00:48:03.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.477+0000 7f0b22ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:03.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.477+0000 7f0b22ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:03.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.477+0000 7f0b22ffd700 10 monclient: dump: 2023-09-30T00:48:03.480 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:03.480 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:03.480 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:03.480 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:03.480 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:03.480 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:03.480 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:03.480 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:03.481 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:03.481 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:03.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b22ffd700 10 monclient: _finish_auth 0 2023-09-30T00:48:03.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b22ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:03.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b22ffd700 1 -- 172.21.15.40:0/2076080530 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1400c7f0 con 0x7f0b2c0af410 2023-09-30T00:48:03.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b30acf700 5 monclient: authenticate success, global_id 4393 2023-09-30T00:48:03.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b30acf700 10 monclient: _renew_subs 2023-09-30T00:48:03.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b30acf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:03.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b30acf700 1 -- 172.21.15.40:0/2076080530 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c13ed60 con 0x7f0b2c0af410 2023-09-30T00:48:03.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b22ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:03.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b22ffd700 1 -- 172.21.15.40:0/2076080530 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1400cb70 con 0x7f0b2c0af410 2023-09-30T00:48:03.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b22ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:03.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b22ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:03.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b22ffd700 10 monclient: dump: 2023-09-30T00:48:03.482 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:03.482 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:03.482 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:03.482 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:03.482 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:03.482 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:03.482 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:03.482 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:03.482 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:03.482 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:03.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b2959a700 4 set_mon_vals no callback set 2023-09-30T00:48:03.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b30acf700 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:03.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b30acf700 10 monclient: _renew_subs 2023-09-30T00:48:03.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b30acf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:03.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b30acf700 1 -- 172.21.15.40:0/2076080530 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c136bc0 con 0x7f0b2c0af410 2023-09-30T00:48:03.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.478+0000 7f0b30acf700 1 librados: init done 2023-09-30T00:48:03.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.479+0000 7f0b30acf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:03.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.479+0000 7f0b30acf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:03.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.479+0000 7f0b30acf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:03.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.479+0000 7f0b30acf700 1 -- 172.21.15.40:0/2076080530 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2c064a00 con 0x7f0b2c0af410 2023-09-30T00:48:03.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.733+0000 7f0b22ffd700 1 -- 172.21.15.40:0/2076080530 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0b14013470 con 0x7f0b2c0af410 2023-09-30T00:48:03.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.734+0000 7f0b22ffd700 1 --2- 172.21.15.40:0/2076080530 >> v2:172.21.15.135:6800/116426 conn(0x7f0b0c068960 0x7f0b0c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:03.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.734+0000 7f0b22ffd700 1 -- 172.21.15.40:0/2076080530 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0b140847b0 con 0x7f0b2c0af410 2023-09-30T00:48:03.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.734+0000 7f0b22ffd700 10 client.4393.objecter ms_dispatch 0x7f0b2c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:03.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.734+0000 7f0b22ffd700 3 client.4393.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:03.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.734+0000 7f0b22ffd700 3 client.4393.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:03.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.734+0000 7f0b22ffd700 20 client.4393.objecter dump_active .. 0 homeless 2023-09-30T00:48:03.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.734+0000 7f0b2a59c700 1 --2- 172.21.15.40:0/2076080530 >> v2:172.21.15.135:6800/116426 conn(0x7f0b0c068960 0x7f0b0c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:03.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.734+0000 7f0b2a59c700 10 monclient: get_auth_request con 0x7f0b0c068960 auth_method 0 2023-09-30T00:48:03.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.735+0000 7f0b22ffd700 10 client.4393.objecter ms_handle_connect 0x7f0b0c068960 2023-09-30T00:48:03.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.735+0000 7f0b2a59c700 1 --2- 172.21.15.40:0/2076080530 >> v2:172.21.15.135:6800/116426 conn(0x7f0b0c068960 0x7f0b0c06ae10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c009d10 tx=0x7f0b1c0091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:03.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.743+0000 7f0b22ffd700 1 -- 172.21.15.40:0/2076080530 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b140520d0 con 0x7f0b2c0af410 2023-09-30T00:48:03.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.743+0000 7f0b22ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:03.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.743+0000 7f0b22ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:03.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.943+0000 7f0b30acf700 1 -- 172.21.15.40:0/2076080530 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0b2c13d430 con 0x7f0b0c068960 2023-09-30T00:48:03.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.946+0000 7f0b22ffd700 1 -- 172.21.15.40:0/2076080530 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48056 (secure 0 0 0) 0x7f0b2c13d430 con 0x7f0b0c068960 2023-09-30T00:48:03.947 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:03.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 10 librados: watch_flush enter 2023-09-30T00:48:03.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 10 librados: watch_flush exit 2023-09-30T00:48:03.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 20 client.4393.objecter shutdown clearing up homeless session... 2023-09-30T00:48:03.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 10 client.4393.objecter successfully canceled tick 2023-09-30T00:48:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 1 -- 172.21.15.40:0/2076080530 >> v2:172.21.15.135:6800/116426 conn(0x7f0b0c068960 msgr2=0x7f0b0c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 1 --2- 172.21.15.40:0/2076080530 >> v2:172.21.15.135:6800/116426 conn(0x7f0b0c068960 0x7f0b0c06ae10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c009d10 tx=0x7f0b1c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 10 monclient: shutdown 2023-09-30T00:48:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 1 -- 172.21.15.40:0/2076080530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b2c0af410 msgr2=0x7f0b2c13cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 1 --2- 172.21.15.40:0/2076080530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b2c0af410 0x7f0b2c13cc30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400ea80 tx=0x7f0b140091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 1 -- 172.21.15.40:0/2076080530 shutdown_connections 2023-09-30T00:48:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 1 --2- 172.21.15.40:0/2076080530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0b2c0b1460 0x7f0b2c13d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 1 --2- 172.21.15.40:0/2076080530 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0b2c0a5fb0 0x7f0b2c0a7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 1 --2- 172.21.15.40:0/2076080530 >> v2:172.21.15.135:6800/116426 conn(0x7f0b0c068960 0x7f0b0c06ae10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 1 --2- 172.21.15.40:0/2076080530 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0b2c0af410 0x7f0b2c13cc30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:03.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 1 -- 172.21.15.40:0/2076080530 >> 172.21.15.40:0/2076080530 conn(0x7f0b2c0631f0 msgr2=0x7f0b2c0b28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:03.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 1 -- 172.21.15.40:0/2076080530 shutdown_connections 2023-09-30T00:48:03.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 1 -- 172.21.15.40:0/2076080530 wait complete. 2023-09-30T00:48:03.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:03.948+0000 7f0b30acf700 1 librados: shutdown 2023-09-30T00:48:03.951 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-09-30T00:48:03.962 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2023-09-30T00:48:03.873883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43464,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931256,"statfs":{"total":773094113280,"available":773049606144,"internally_reserved":0,"allocated":2506752,"data_stored":1383352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":18,"num_write":18,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.230939"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.785504+0000","last_change":"2023-09-30T00:47:54.691712+0000","last_active":"2023-09-30T00:47:54.785504+0000","last_peered":"2023-09-30T00:47:54.785504+0000","last_clean":"2023-09-30T00:47:54.785504+0000","last_became_active":"2023-09-30T00:47:54.672127+0000","last_became_peered":"2023-09-30T00:47:54.672127+0000","last_unstale":"2023-09-30T00:47:54.785504+0000","last_undegraded":"2023-09-30T00:47:54.785504+0000","last_fullsized":"2023-09-30T00:47:54.785504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:32.704584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.160955+0000","last_change":"2023-09-30T00:47:54.749019+0000","last_active":"2023-09-30T00:47:55.160955+0000","last_peered":"2023-09-30T00:47:55.160955+0000","last_clean":"2023-09-30T00:47:55.160955+0000","last_became_active":"2023-09-30T00:47:54.702773+0000","last_became_peered":"2023-09-30T00:47:54.702773+0000","last_unstale":"2023-09-30T00:47:55.160955+0000","last_undegraded":"2023-09-30T00:47:55.160955+0000","last_fullsized":"2023-09-30T00:47:55.160955+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:07:10.081341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.809132+0000","last_change":"2023-09-30T00:47:54.157108+0000","last_active":"2023-09-30T00:47:54.809132+0000","last_peered":"2023-09-30T00:47:54.809132+0000","last_clean":"2023-09-30T00:47:54.809132+0000","last_became_active":"2023-09-30T00:47:54.139464+0000","last_became_peered":"2023-09-30T00:47:54.139464+0000","last_unstale":"2023-09-30T00:47:54.809132+0000","last_undegraded":"2023-09-30T00:47:54.809132+0000","last_fullsized":"2023-09-30T00:47:54.809132+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:42:38.117935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.795839+0000","last_change":"2023-09-30T00:47:53.993668+0000","last_active":"2023-09-30T00:47:54.795839+0000","last_peered":"2023-09-30T00:47:54.795839+0000","last_clean":"2023-09-30T00:47:54.795839+0000","last_became_active":"2023-09-30T00:47:53.951882+0000","last_became_peered":"2023-09-30T00:47:53.951882+0000","last_unstale":"2023-09-30T00:47:54.795839+0000","last_undegraded":"2023-09-30T00:47:54.795839+0000","last_fullsized":"2023-09-30T00:47:54.795839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:15:02.278001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'77","reported_seq":114,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:47:57.719324+0000","last_change":"2023-09-30T00:47:47.347565+0000","last_active":"2023-09-30T00:47:57.719324+0000","last_peered":"2023-09-30T00:47:57.719324+0000","last_clean":"2023-09-30T00:47:57.719324+0000","last_became_active":"2023-09-30T00:47:46.406306+0000","last_became_peered":"2023-09-30T00:47:46.406306+0000","last_unstale":"2023-09-30T00:47:57.719324+0000","last_undegraded":"2023-09-30T00:47:57.719324+0000","last_fullsized":"2023-09-30T00:47:57.719324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:40.867840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:40.867840+0000","last_clean_scrub_stamp":"2023-09-30T00:47:40.867840+0000","objects_scrubbed":0,"log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:17:21.219288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.780151+0000","last_change":"2023-09-30T00:47:54.729272+0000","last_active":"2023-09-30T00:47:54.780151+0000","last_peered":"2023-09-30T00:47:54.780151+0000","last_clean":"2023-09-30T00:47:54.780151+0000","last_became_active":"2023-09-30T00:47:54.710752+0000","last_became_peered":"2023-09-30T00:47:54.710752+0000","last_unstale":"2023-09-30T00:47:54.780151+0000","last_undegraded":"2023-09-30T00:47:54.780151+0000","last_fullsized":"2023-09-30T00:47:54.780151+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:59:21.956005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.239361+0000","last_change":"2023-09-30T00:47:53.773125+0000","last_active":"2023-09-30T00:47:55.239361+0000","last_peered":"2023-09-30T00:47:55.239361+0000","last_clean":"2023-09-30T00:47:55.239361+0000","last_became_active":"2023-09-30T00:47:53.761460+0000","last_became_peered":"2023-09-30T00:47:53.761460+0000","last_unstale":"2023-09-30T00:47:55.239361+0000","last_undegraded":"2023-09-30T00:47:55.239361+0000","last_fullsized":"2023-09-30T00:47:55.239361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:55.538707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.791094+0000","last_change":"2023-09-30T00:47:54.294197+0000","last_active":"2023-09-30T00:47:54.791094+0000","last_peered":"2023-09-30T00:47:54.791094+0000","last_clean":"2023-09-30T00:47:54.791094+0000","last_became_active":"2023-09-30T00:47:54.279706+0000","last_became_peered":"2023-09-30T00:47:54.279706+0000","last_unstale":"2023-09-30T00:47:54.791094+0000","last_undegraded":"2023-09-30T00:47:54.791094+0000","last_fullsized":"2023-09-30T00:47:54.791094+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:24:53.339591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.240181+0000","last_change":"2023-09-30T00:47:54.352713+0000","last_active":"2023-09-30T00:47:55.240181+0000","last_peered":"2023-09-30T00:47:55.240181+0000","last_clean":"2023-09-30T00:47:55.240181+0000","last_became_active":"2023-09-30T00:47:54.305507+0000","last_became_peered":"2023-09-30T00:47:54.305507+0000","last_unstale":"2023-09-30T00:47:55.240181+0000","last_undegraded":"2023-09-30T00:47:55.240181+0000","last_fullsized":"2023-09-30T00:47:55.240181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:21:08.873800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345928,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2550000000000008}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.238}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.448}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.033999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.835000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.471}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.646999999999998}]}]},{"osd":2,"up_from":20,"seq":85899345927,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.42}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.690999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.763999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.837}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.577999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.646000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.442}]}]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":614400,"data_stored":471299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.396000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5039999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.419}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.076000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.8010000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.814}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.652999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.098000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.222000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.800999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.609000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.375}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.542999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.617999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.0739999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3190000000000008}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.356000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.914999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.492000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.476000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.484999999999999}]}]},{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.6199999999999992}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.605}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.693999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.975}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.768999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.376999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.343}]}]},{"osd":0,"up_from":20,"seq":85899345928,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.586}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.856}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.518000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.780999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.098000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.289999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.065999999999999}]}]},{"osd":1,"up_from":19,"seq":81604378631,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6044,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365796,"statfs":{"total":96636764160,"available":96630575104,"internally_reserved":0,"allocated":614400,"data_stored":471299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.711}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.259}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.108000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.56}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.704000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.382999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.912000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:48:03.964 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T00:48:03.964 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-30T00:48:03.964 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-30T00:48:03.964 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:04.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.094+0000 7f8f880b9700 10 monclient: get_monmap_and_config 2023-09-30T00:48:04.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.094+0000 7f8f880b9700 10 monclient: build_initial_monmap 2023-09-30T00:48:04.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.094+0000 7f8f880b9700 10 monclient: monmap: 2023-09-30T00:48:04.095 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:04.095 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:04.095 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:04.095456+0000 2023-09-30T00:48:04.095 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:04.095456+0000 2023-09-30T00:48:04.095 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:04.095 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:04.095 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:04.096 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:04.096 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:04.096 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:04.096 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:04.096 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 1 Processor -- start 2023-09-30T00:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 1 -- start start 2023-09-30T00:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 10 monclient: init 2023-09-30T00:48:04.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2023-09-30T00:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f800b1460 0x7f8f800b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 10 monclient(hunting): picked mon.noname-b con 0x7f8f800b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f800a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f8013ac20 0x7f8f8013b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 10 monclient(hunting): picked mon.noname-a con 0x7f8f8013ac20 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f800a5b20 con 0x7f8f800a5fb0 2023-09-30T00:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f800a71f0 con 0x7f8f8013ac20 2023-09-30T00:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:04.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f800a74d0 con 0x7f8f800b1460 2023-09-30T00:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.095+0000 7f8f880b9700 10 monclient(hunting): authenticate will time out at 5345.396969s 2023-09-30T00:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.096+0000 7f8f86656700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f8013ac20 0x7f8f8013b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.096+0000 7f8f85654700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f8f800a5fb0 0x7f8f80141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:58136/0 (socket says 172.21.15.40:58136) 2023-09-30T00:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.096+0000 7f8f85654700 1 -- 172.21.15.40:0/548306634 learned_addr learned my addr 172.21.15.40:0/548306634 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.096+0000 7f8f85e55700 1 --2- 172.21.15.40:0/548306634 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f800b1460 0x7f8f800b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.097+0000 7f8f86656700 10 monclient(hunting): get_auth_request con 0x7f8f8013ac20 auth_method 0 2023-09-30T00:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.097+0000 7f8f86656700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.097+0000 7f8f86656700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.097+0000 7f8f86656700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.097+0000 7f8f85e55700 10 monclient(hunting): get_auth_request con 0x7f8f800b1460 auth_method 0 2023-09-30T00:48:04.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.097+0000 7f8f85e55700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:04.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.097+0000 7f8f85e55700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:04.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.097+0000 7f8f85e55700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.097+0000 7f8f86656700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.097+0000 7f8f86656700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.097+0000 7f8f86656700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.097+0000 7f8f85e55700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.097+0000 7f8f85e55700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.097+0000 7f8f85e55700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.099+0000 7f8f84e53700 1 -- 172.21.15.40:0/548306634 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871345896 0 0) 0x7f8f800a5b20 con 0x7f8f800a5fb0 2023-09-30T00:48:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.099+0000 7f8f84e53700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.099+0000 7f8f84e53700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.099+0000 7f8f84e53700 10 monclient(hunting): my global_id is 4396 2023-09-30T00:48:04.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.099+0000 7f8f84e53700 1 -- 172.21.15.40:0/548306634 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f6c003570 con 0x7f8f800a5fb0 2023-09-30T00:48:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.099+0000 7f8f86656700 10 monclient(hunting): handle_auth_done global_id 4368 payload 274 2023-09-30T00:48:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.099+0000 7f8f86656700 1 -- 172.21.15.40:0/548306634 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f800b1460 msgr2=0x7f8f800b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.099+0000 7f8f86656700 1 --2- 172.21.15.40:0/548306634 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f800b1460 0x7f8f800b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.099+0000 7f8f86656700 1 -- 172.21.15.40:0/548306634 >> v1:172.21.15.135:6789/0 conn(0x7f8f800a5fb0 legacy=0x7f8f80141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.099+0000 7f8f86656700 10 monclient: _finish_hunting 0 2023-09-30T00:48:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.099+0000 7f8f86656700 1 monclient: found mon.noname-a 2023-09-30T00:48:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.099+0000 7f8f86656700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.099+0000 7f8f86656700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:04.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.099+0000 7f8f86656700 1 -- 172.21.15.40:0/548306634 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f800a63c0 con 0x7f8f8013ac20 2023-09-30T00:48:04.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.102+0000 7f8f86656700 1 --2- 172.21.15.40:0/548306634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8013ac20 0x7f8f8013b0a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009e60 tx=0x7f8f78009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0d7268516f4ea8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:04.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f84e53700 1 -- 172.21.15.40:0/548306634 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7800e070 con 0x7f8f8013ac20 2023-09-30T00:48:04.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f84e53700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:04.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f84e53700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:04.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f84e53700 10 monclient: dump: 2023-09-30T00:48:04.108 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:04.109 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:04.109 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:04.109 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:04.109 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:04.109 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:04.109 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:04.109 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:04.109 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:04.109 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:04.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f84e53700 10 monclient: _finish_auth 0 2023-09-30T00:48:04.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f84e53700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:04.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f84e53700 1 -- 172.21.15.40:0/548306634 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7800bab0 con 0x7f8f8013ac20 2023-09-30T00:48:04.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f880b9700 5 monclient: authenticate success, global_id 4368 2023-09-30T00:48:04.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f880b9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:04.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f84e53700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:04.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f880b9700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:04.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f84e53700 1 -- 172.21.15.40:0/548306634 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7800be70 con 0x7f8f8013ac20 2023-09-30T00:48:04.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f84e53700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:04.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f880b9700 4 set_mon_vals no callback set 2023-09-30T00:48:04.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f84e53700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:04.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.106+0000 7f8f84e53700 10 monclient: dump: 2023-09-30T00:48:04.110 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:04.110 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:04.110 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:04.110 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:04.111 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:04.111 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:04.111 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:04.111 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:04.111 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:04.111 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:04.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.107+0000 7f8f880b9700 10 monclient: shutdown 2023-09-30T00:48:04.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.107+0000 7f8f880b9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:04.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.107+0000 7f8f880b9700 1 -- 172.21.15.40:0/548306634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8013ac20 msgr2=0x7f8f8013b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:04.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.107+0000 7f8f880b9700 1 --2- 172.21.15.40:0/548306634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8013ac20 0x7f8f8013b0a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009e60 tx=0x7f8f78009780 comp rx=0 tx=0).stop 2023-09-30T00:48:04.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.107+0000 7f8f880b9700 1 -- 172.21.15.40:0/548306634 shutdown_connections 2023-09-30T00:48:04.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.107+0000 7f8f880b9700 1 --2- 172.21.15.40:0/548306634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8013ac20 0x7f8f8013b0a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:04.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.107+0000 7f8f880b9700 1 --2- 172.21.15.40:0/548306634 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f800b1460 0x7f8f800b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:04.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.107+0000 7f8f880b9700 1 -- 172.21.15.40:0/548306634 >> 172.21.15.40:0/548306634 conn(0x7f8f800631f0 msgr2=0x7f8f800635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:04.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.107+0000 7f8f880b9700 1 -- 172.21.15.40:0/548306634 shutdown_connections 2023-09-30T00:48:04.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.107+0000 7f8f880b9700 1 -- 172.21.15.40:0/548306634 wait complete. 2023-09-30T00:48:04.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.108+0000 7f8f880b9700 10 monclient: build_initial_monmap 2023-09-30T00:48:04.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.108+0000 7f8f880b9700 10 monclient: monmap: 2023-09-30T00:48:04.115 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:04.115 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:04.115 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:04.115 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:04.116 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:04.116 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:04.116 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:04.116 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:04.116 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:04.116 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:04.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.108+0000 7f8f880b9700 1 librados: starting msgr at 2023-09-30T00:48:04.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.108+0000 7f8f880b9700 1 librados: starting objecter 2023-09-30T00:48:04.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.108+0000 7f8f880b9700 1 Processor -- start 2023-09-30T00:48:04.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 1 -- start start 2023-09-30T00:48:04.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 1 librados: setting wanted keys 2023-09-30T00:48:04.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 1 librados: calling monclient init 2023-09-30T00:48:04.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 10 monclient: init 2023-09-30T00:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f800a5fb0 0x7f8f800aa7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 10 monclient(hunting): picked mon.noname-c con 0x7f8f800a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f800b1460 0x7f8f8013e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 10 monclient(hunting): picked mon.noname-b con 0x7f8f800b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8013ac20 0x7f8f80150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 10 monclient(hunting): picked mon.noname-a con 0x7f8f8013ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f800aa350 con 0x7f8f8013ac20 2023-09-30T00:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:04.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f800ab9e0 con 0x7f8f800b1460 2023-09-30T00:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f800abc50 con 0x7f8f800a5fb0 2023-09-30T00:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.109+0000 7f8f880b9700 10 monclient(hunting): authenticate will time out at 5345.410890s 2023-09-30T00:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.110+0000 7f8f86656700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8013ac20 0x7f8f80150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.110+0000 7f8f85654700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f800b1460 0x7f8f8013e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.110+0000 7f8f85e55700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f800a5fb0 0x7f8f800aa7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.110+0000 7f8f86656700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8013ac20 0x7f8f80150f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52460/0 (socket says 172.21.15.40:52460) 2023-09-30T00:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.110+0000 7f8f86656700 1 -- 172.21.15.40:0/2816665816 learned_addr learned my addr 172.21.15.40:0/2816665816 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.110+0000 7f8f86656700 10 monclient(hunting): get_auth_request con 0x7f8f8013ac20 auth_method 0 2023-09-30T00:48:04.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.110+0000 7f8f86656700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:04.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.110+0000 7f8f86656700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:04.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.110+0000 7f8f86656700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:04.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.110+0000 7f8f85654700 10 monclient(hunting): get_auth_request con 0x7f8f800b1460 auth_method 0 2023-09-30T00:48:04.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.110+0000 7f8f85654700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:04.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.110+0000 7f8f85654700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:04.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.110+0000 7f8f85654700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:04.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.111+0000 7f8f85e55700 10 monclient(hunting): get_auth_request con 0x7f8f800a5fb0 auth_method 0 2023-09-30T00:48:04.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.111+0000 7f8f85e55700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:04.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.111+0000 7f8f85e55700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:04.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.111+0000 7f8f85e55700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:04.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.111+0000 7f8f86656700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:04.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.111+0000 7f8f86656700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:04.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.111+0000 7f8f86656700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.111+0000 7f8f85654700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.111+0000 7f8f85654700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.111+0000 7f8f85654700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.111+0000 7f8f85e55700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.111+0000 7f8f85e55700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.111+0000 7f8f85e55700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.114+0000 7f8f86656700 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2023-09-30T00:48:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.114+0000 7f8f86656700 1 -- 172.21.15.40:0/2816665816 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f800a5fb0 msgr2=0x7f8f800aa7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.114+0000 7f8f86656700 1 --2- 172.21.15.40:0/2816665816 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f800a5fb0 0x7f8f800aa7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.114+0000 7f8f86656700 1 -- 172.21.15.40:0/2816665816 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f800b1460 msgr2=0x7f8f8013e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.114+0000 7f8f86656700 1 --2- 172.21.15.40:0/2816665816 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f800b1460 0x7f8f8013e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:04.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.114+0000 7f8f86656700 10 monclient: _finish_hunting 0 2023-09-30T00:48:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.114+0000 7f8f86656700 1 monclient: found mon.noname-a 2023-09-30T00:48:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.114+0000 7f8f86656700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.114+0000 7f8f86656700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.114+0000 7f8f86656700 1 -- 172.21.15.40:0/2816665816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f78009a20 con 0x7f8f8013ac20 2023-09-30T00:48:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.117+0000 7f8f86656700 1 --2- 172.21.15.40:0/2816665816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8013ac20 0x7f8f80150f50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8f8006f5e0 tx=0x7f8f78019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.117+0000 7f8f75ffb700 10 client.?.objecter ms_handle_connect 0x7f8f8013ac20 2023-09-30T00:48:04.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.117+0000 7f8f75ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f75ffb700 1 -- 172.21.15.40:0/2816665816 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f78012e10 con 0x7f8f8013ac20 2023-09-30T00:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f75ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f75ffb700 10 monclient: dump: 2023-09-30T00:48:04.123 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f75ffb700 10 monclient: _finish_auth 0 2023-09-30T00:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f75ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f75ffb700 1 -- 172.21.15.40:0/2816665816 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f78027470 con 0x7f8f8013ac20 2023-09-30T00:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f75ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f75ffb700 1 -- 172.21.15.40:0/2816665816 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f78027830 con 0x7f8f8013ac20 2023-09-30T00:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f880b9700 5 monclient: authenticate success, global_id 4371 2023-09-30T00:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f880b9700 10 monclient: _renew_subs 2023-09-30T00:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f880b9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f880b9700 1 -- 172.21.15.40:0/2816665816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f800a8a20 con 0x7f8f8013ac20 2023-09-30T00:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f75ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f75ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f77fff700 4 set_mon_vals no callback set 2023-09-30T00:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.121+0000 7f8f75ffb700 10 monclient: dump: 2023-09-30T00:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.122+0000 7f8f880b9700 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.122+0000 7f8f880b9700 10 monclient: _renew_subs 2023-09-30T00:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.122+0000 7f8f880b9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.122+0000 7f8f880b9700 1 -- 172.21.15.40:0/2816665816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f80151490 con 0x7f8f8013ac20 2023-09-30T00:48:04.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.122+0000 7f8f880b9700 1 librados: init done 2023-09-30T00:48:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.123+0000 7f8f880b9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.123+0000 7f8f880b9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.123+0000 7f8f880b9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:04.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.123+0000 7f8f880b9700 1 -- 172.21.15.40:0/2816665816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f800aa500 con 0x7f8f8013ac20 2023-09-30T00:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.270+0000 7f8f75ffb700 1 -- 172.21.15.40:0/2816665816 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8f78024070 con 0x7f8f8013ac20 2023-09-30T00:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.270+0000 7f8f75ffb700 1 --2- 172.21.15.40:0/2816665816 >> v2:172.21.15.135:6800/116426 conn(0x7f8f64068890 0x7f8f6406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.270+0000 7f8f75ffb700 1 -- 172.21.15.40:0/2816665816 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8f78086aa0 con 0x7f8f8013ac20 2023-09-30T00:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.270+0000 7f8f75ffb700 10 client.4371.objecter ms_dispatch 0x7f8f80001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.270+0000 7f8f75ffb700 3 client.4371.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:04.271 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.270+0000 7f8f75ffb700 3 client.4371.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.270+0000 7f8f75ffb700 20 client.4371.objecter dump_active .. 0 homeless 2023-09-30T00:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.270+0000 7f8f85e55700 1 --2- 172.21.15.40:0/2816665816 >> v2:172.21.15.135:6800/116426 conn(0x7f8f64068890 0x7f8f6406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.270+0000 7f8f85e55700 10 monclient: get_auth_request con 0x7f8f64068890 auth_method 0 2023-09-30T00:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.271+0000 7f8f75ffb700 10 client.4371.objecter ms_handle_connect 0x7f8f64068890 2023-09-30T00:48:04.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.271+0000 7f8f85e55700 1 --2- 172.21.15.40:0/2816665816 >> v2:172.21.15.135:6800/116426 conn(0x7f8f64068890 0x7f8f6406ad40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c009db0 tx=0x7f8f7c0091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:04.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.428+0000 7f8f75ffb700 1 -- 172.21.15.40:0/2816665816 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8f78054340 con 0x7f8f8013ac20 2023-09-30T00:48:04.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.428+0000 7f8f75ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:04.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.428+0000 7f8f75ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:04.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.685+0000 7f8f880b9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:04.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.685+0000 7f8f880b9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:04.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.685+0000 7f8f880b9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:04.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.685+0000 7f8f880b9700 1 -- 172.21.15.40:0/2816665816 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8f800aa500 con 0x7f8f8013ac20 2023-09-30T00:48:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.697+0000 7f8f75ffb700 1 -- 172.21.15.40:0/2816665816 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8f800aa500 con 0x7f8f8013ac20 2023-09-30T00:48:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.697+0000 7f8f75ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.697+0000 7f8f75ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:04.698 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:04.698 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:04.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 10 librados: watch_flush enter 2023-09-30T00:48:04.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 10 librados: watch_flush exit 2023-09-30T00:48:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 20 client.4371.objecter shutdown clearing up homeless session... 2023-09-30T00:48:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 10 client.4371.objecter successfully canceled tick 2023-09-30T00:48:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 1 -- 172.21.15.40:0/2816665816 >> v2:172.21.15.135:6800/116426 conn(0x7f8f64068890 msgr2=0x7f8f6406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 1 --2- 172.21.15.40:0/2816665816 >> v2:172.21.15.135:6800/116426 conn(0x7f8f64068890 0x7f8f6406ad40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c009db0 tx=0x7f8f7c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 10 monclient: shutdown 2023-09-30T00:48:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 1 -- 172.21.15.40:0/2816665816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8013ac20 msgr2=0x7f8f80150f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 1 --2- 172.21.15.40:0/2816665816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8013ac20 0x7f8f80150f50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8f8006f5e0 tx=0x7f8f78019000 comp rx=0 tx=0).stop 2023-09-30T00:48:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 1 -- 172.21.15.40:0/2816665816 shutdown_connections 2023-09-30T00:48:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 1 --2- 172.21.15.40:0/2816665816 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8013ac20 0x7f8f80150f50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:04.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 1 --2- 172.21.15.40:0/2816665816 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8f800a5fb0 0x7f8f800aa7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:04.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 1 --2- 172.21.15.40:0/2816665816 >> v2:172.21.15.135:6800/116426 conn(0x7f8f64068890 0x7f8f6406ad40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:04.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 1 --2- 172.21.15.40:0/2816665816 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8f800b1460 0x7f8f8013e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:04.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 1 -- 172.21.15.40:0/2816665816 >> 172.21.15.40:0/2816665816 conn(0x7f8f800631f0 msgr2=0x7f8f800b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:04.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 1 -- 172.21.15.40:0/2816665816 shutdown_connections 2023-09-30T00:48:04.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 1 -- 172.21.15.40:0/2816665816 wait complete. 2023-09-30T00:48:04.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:04.700+0000 7f8f880b9700 1 librados: shutdown 2023-09-30T00:48:07.706 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:01:43.417 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:07.715 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:07.736 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:01:43.447 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:07.764 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:43.451 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:07.768 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:43.455 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:07.770 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:01:43.482 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:07.773 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:01:43.485 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:07.861 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:01:43.572 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:07.864 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:01:43.551 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:07.873 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:01:43.585 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:07.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.878+0000 7fdf84ca6700 10 monclient: get_monmap_and_config 2023-09-30T00:48:07.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.878+0000 7fdf84ca6700 10 monclient: build_initial_monmap 2023-09-30T00:48:07.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.878+0000 7fdf84ca6700 10 monclient: monmap: 2023-09-30T00:48:07.879 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:07.879 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:07.879 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:07.879196+0000 2023-09-30T00:48:07.879 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:07.879196+0000 2023-09-30T00:48:07.879 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:07.879 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:07.879 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:07.879 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:07.879 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:07.879 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:07.879 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:07.880 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.878+0000 7fdf84ca6700 1 Processor -- start 2023-09-30T00:48:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.878+0000 7fdf84ca6700 1 -- start start 2023-09-30T00:48:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.878+0000 7fdf84ca6700 10 monclient: init 2023-09-30T00:48:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.879+0000 7fdf84ca6700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.879+0000 7fdf84ca6700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-09-30T00:48:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.879+0000 7fdf84ca6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf800b1460 0x7fdf800b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.879+0000 7fdf84ca6700 10 monclient(hunting): picked mon.noname-a con 0x7fdf800b1460 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.879+0000 7fdf84ca6700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdf800a5fb0 0x7fdf80141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.879+0000 7fdf84ca6700 10 monclient(hunting): picked mon.noname-c con 0x7fdf800a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.879+0000 7fdf84ca6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf800af410 addr v1:172.21.15.40:6789/0 2023-09-30T00:48:07.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.879+0000 7fdf84ca6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf800a71f0 con 0x7fdf800af410 2023-09-30T00:48:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.879+0000 7fdf84ca6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.879+0000 7fdf84ca6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf800a74d0 con 0x7fdf800b1460 2023-09-30T00:48:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.879+0000 7fdf84ca6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.879+0000 7fdf84ca6700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf800a63c0 con 0x7fdf800a5fb0 2023-09-30T00:48:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.879+0000 7fdf84ca6700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:07.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.879+0000 7fdf84ca6700 10 monclient(hunting): authenticate will time out at 5349.180845s 2023-09-30T00:48:07.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.892+0000 7fdf7e59c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf800b1460 0x7fdf800b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.902+0000 7fdf7e59c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf800b1460 0x7fdf800b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52476/0 (socket says 172.21.15.40:52476) 2023-09-30T00:48:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.902+0000 7fdf7e59c700 1 -- 172.21.15.40:0/92413752 learned_addr learned my addr 172.21.15.40:0/92413752 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:07.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.902+0000 7fdf7e59c700 10 monclient(hunting): get_auth_request con 0x7fdf800b1460 auth_method 0 2023-09-30T00:48:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.902+0000 7fdf7e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.902+0000 7fdf7e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:07.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.902+0000 7fdf7e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:07.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.906+0000 7fdf7dd9b700 1 --2- 172.21.15.40:0/92413752 >> v2:172.21.15.135:3300/0 conn(0x7fdf800a5fb0 0x7fdf80141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:07.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.907+0000 7fdf7dd9b700 10 monclient(hunting): get_auth_request con 0x7fdf800a5fb0 auth_method 0 2023-09-30T00:48:07.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.907+0000 7fdf7dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:07.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.907+0000 7fdf7dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:07.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.907+0000 7fdf7dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:07.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.911+0000 7fdf7dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:07.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.911+0000 7fdf7dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:07.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.911+0000 7fdf7dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:07.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.914+0000 7fdf7e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:07.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.914+0000 7fdf7e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:07.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.914+0000 7fdf7e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.915+0000 7fdf7dd9b700 10 monclient(hunting): handle_auth_done global_id 4402 payload 274 2023-09-30T00:48:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.915+0000 7fdf7dd9b700 1 -- 172.21.15.40:0/92413752 >> v2:172.21.15.40:3300/0 conn(0x7fdf800b1460 msgr2=0x7fdf800b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.915+0000 7fdf7dd9b700 1 --2- 172.21.15.40:0/92413752 >> v2:172.21.15.40:3300/0 conn(0x7fdf800b1460 0x7fdf800b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.915+0000 7fdf7dd9b700 1 -- 172.21.15.40:0/92413752 >> v1:172.21.15.40:6789/0 conn(0x7fdf800af410 legacy=0x7fdf800af870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.915+0000 7fdf7dd9b700 10 monclient: _finish_hunting 0 2023-09-30T00:48:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.915+0000 7fdf7dd9b700 1 monclient: found mon.noname-c 2023-09-30T00:48:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.915+0000 7fdf7dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.915+0000 7fdf7dd9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:48:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.915+0000 7fdf7dd9b700 1 -- 172.21.15.40:0/92413752 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf800a56d0 con 0x7fdf800a5fb0 2023-09-30T00:48:07.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.926+0000 7fdf7dd9b700 1 --2- 172.21.15.40:0/92413752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf800a5fb0 0x7fdf80141730 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdf740098e0 tx=0x7fdf740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71e8de3c170aae6e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:07.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.952+0000 7fdf7d59a700 1 -- 172.21.15.40:0/92413752 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74011630 con 0x7fdf800a5fb0 2023-09-30T00:48:07.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.952+0000 7fdf7d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:07.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.952+0000 7fdf7d59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:07.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.952+0000 7fdf7d59a700 10 monclient: dump: 2023-09-30T00:48:07.953 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:07.953 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:07.953 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:07.953 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:07.953 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:07.954 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:07.954 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:07.954 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:07.954 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:07.954 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:07.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.952+0000 7fdf7d59a700 10 monclient: _finish_auth 0 2023-09-30T00:48:07.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.952+0000 7fdf7d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:07.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.952+0000 7fdf84ca6700 5 monclient: authenticate success, global_id 4402 2023-09-30T00:48:07.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:07.952+0000 7fdf84ca6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:08.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.013+0000 7fdf7d59a700 1 -- 172.21.15.40:0/92413752 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf74011790 con 0x7fdf800a5fb0 2023-09-30T00:48:08.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.013+0000 7fdf7d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:08.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.013+0000 7fdf7d59a700 1 -- 172.21.15.40:0/92413752 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74011b30 con 0x7fdf800a5fb0 2023-09-30T00:48:08.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.013+0000 7fdf84ca6700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:08.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.013+0000 7fdf84ca6700 4 set_mon_vals no callback set 2023-09-30T00:48:08.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.013+0000 7fdf7d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:08.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.013+0000 7fdf7d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:08.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.013+0000 7fdf7d59a700 10 monclient: dump: 2023-09-30T00:48:08.014 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:08.014 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:08.014 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:08.015 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:08.015 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:08.015 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:08.015 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:08.015 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:08.015 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:08.015 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:08.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.013+0000 7fdf84ca6700 10 monclient: shutdown 2023-09-30T00:48:08.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.013+0000 7fdf84ca6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:08.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.013+0000 7fdf84ca6700 1 -- 172.21.15.40:0/92413752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf800a5fb0 msgr2=0x7fdf80141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:08.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.013+0000 7fdf84ca6700 1 --2- 172.21.15.40:0/92413752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf800a5fb0 0x7fdf80141730 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdf740098e0 tx=0x7fdf740091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:08.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.014+0000 7fdf84ca6700 1 -- 172.21.15.40:0/92413752 shutdown_connections 2023-09-30T00:48:08.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.014+0000 7fdf84ca6700 1 --2- 172.21.15.40:0/92413752 >> v2:172.21.15.40:3300/0 conn(0x7fdf800b1460 0x7fdf800b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:08.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.014+0000 7fdf84ca6700 1 --2- 172.21.15.40:0/92413752 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf800a5fb0 0x7fdf80141730 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:08.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.014+0000 7fdf84ca6700 1 -- 172.21.15.40:0/92413752 >> 172.21.15.40:0/92413752 conn(0x7fdf800631f0 msgr2=0x7fdf800635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:08.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.014+0000 7fdf84ca6700 1 -- 172.21.15.40:0/92413752 shutdown_connections 2023-09-30T00:48:08.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.014+0000 7fdf84ca6700 1 -- 172.21.15.40:0/92413752 wait complete. 2023-09-30T00:48:08.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.015+0000 7fdf84ca6700 10 monclient: build_initial_monmap 2023-09-30T00:48:08.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.015+0000 7fdf84ca6700 10 monclient: monmap: 2023-09-30T00:48:08.019 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:08.019 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:08.019 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:08.019 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:08.019 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:08.019 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:08.019 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:08.019 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:08.019 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:08.020 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:08.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.015+0000 7fdf84ca6700 1 librados: starting msgr at 2023-09-30T00:48:08.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.015+0000 7fdf84ca6700 1 librados: starting objecter 2023-09-30T00:48:08.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.015+0000 7fdf84ca6700 1 Processor -- start 2023-09-30T00:48:08.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.015+0000 7fdf84ca6700 1 -- start start 2023-09-30T00:48:08.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.015+0000 7fdf84ca6700 1 librados: setting wanted keys 2023-09-30T00:48:08.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.015+0000 7fdf84ca6700 1 librados: calling monclient init 2023-09-30T00:48:08.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.015+0000 7fdf84ca6700 10 monclient: init 2023-09-30T00:48:08.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:08.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:48:08.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf800a5fb0 0x7fdf8013d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:08.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 10 monclient(hunting): picked mon.noname-a con 0x7fdf800a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:08.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf800af410 0x7fdf800a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:08.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 10 monclient(hunting): picked mon.noname-c con 0x7fdf800af410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:08.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf800b1460 0x7fdf80151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:08.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 10 monclient(hunting): picked mon.noname-b con 0x7fdf800b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:08.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:08.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf80151450 con 0x7fdf800a5fb0 2023-09-30T00:48:08.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:08.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf801516f0 con 0x7fdf800b1460 2023-09-30T00:48:08.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:08.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf800a7ba0 con 0x7fdf800af410 2023-09-30T00:48:08.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:08.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.016+0000 7fdf84ca6700 10 monclient(hunting): authenticate will time out at 5349.317584s 2023-09-30T00:48:08.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.017+0000 7fdf7e59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf800a5fb0 0x7fdf8013d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:08.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.022+0000 7fdf7e59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf800a5fb0 0x7fdf8013d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52478/0 (socket says 172.21.15.40:52478) 2023-09-30T00:48:08.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.022+0000 7fdf7e59c700 1 -- 172.21.15.40:0/2886220718 learned_addr learned my addr 172.21.15.40:0/2886220718 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:08.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.022+0000 7fdf7e59c700 10 monclient(hunting): get_auth_request con 0x7fdf800a5fb0 auth_method 0 2023-09-30T00:48:08.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.022+0000 7fdf7e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:08.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.022+0000 7fdf7e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:08.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.022+0000 7fdf7e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:08.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.029+0000 7fdf7ed9d700 1 --2- 172.21.15.40:0/2886220718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf800b1460 0x7fdf80151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:08.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.030+0000 7fdf7dd9b700 1 --2- 172.21.15.40:0/2886220718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf800af410 0x7fdf800a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:08.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.030+0000 7fdf7e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:08.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.030+0000 7fdf7e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:08.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.030+0000 7fdf7e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:08.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.033+0000 7fdf7ed9d700 10 monclient(hunting): get_auth_request con 0x7fdf800b1460 auth_method 0 2023-09-30T00:48:08.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.033+0000 7fdf7ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:08.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.033+0000 7fdf7ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:08.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.033+0000 7fdf7ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:08.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.038+0000 7fdf7e59c700 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2023-09-30T00:48:08.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.038+0000 7fdf7e59c700 1 -- 172.21.15.40:0/2886220718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf800af410 msgr2=0x7fdf800a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:08.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.038+0000 7fdf7e59c700 1 --2- 172.21.15.40:0/2886220718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf800af410 0x7fdf800a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:08.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.038+0000 7fdf7e59c700 1 -- 172.21.15.40:0/2886220718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf800b1460 msgr2=0x7fdf80151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:08.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.038+0000 7fdf7e59c700 1 --2- 172.21.15.40:0/2886220718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf800b1460 0x7fdf80151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:08.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.038+0000 7fdf7e59c700 10 monclient: _finish_hunting 0 2023-09-30T00:48:08.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.038+0000 7fdf7e59c700 1 monclient: found mon.noname-a 2023-09-30T00:48:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.038+0000 7fdf7e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.038+0000 7fdf7e59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.038+0000 7fdf7e59c700 1 -- 172.21.15.40:0/2886220718 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf74009420 con 0x7fdf800a5fb0 2023-09-30T00:48:08.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.101+0000 7fdf7e59c700 1 --2- 172.21.15.40:0/2886220718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf800a5fb0 0x7fdf8013d5a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000e950 tx=0x7fdf70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:08.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.101+0000 7fdf6e7fc700 10 client.?.objecter ms_handle_connect 0x7fdf800a5fb0 2023-09-30T00:48:08.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.101+0000 7fdf6e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:08.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf6e7fc700 1 -- 172.21.15.40:0/2886220718 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7000c850 con 0x7fdf800a5fb0 2023-09-30T00:48:08.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf6e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:08.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf6e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf6e7fc700 10 monclient: dump: 2023-09-30T00:48:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:08.115 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf6e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf6e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf6e7fc700 1 -- 172.21.15.40:0/2886220718 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7000c9b0 con 0x7fdf800a5fb0 2023-09-30T00:48:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf6e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf6e7fc700 1 -- 172.21.15.40:0/2886220718 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7000cd70 con 0x7fdf800a5fb0 2023-09-30T00:48:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf84ca6700 5 monclient: authenticate success, global_id 4377 2023-09-30T00:48:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf6e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf6e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf6e7fc700 10 monclient: dump: 2023-09-30T00:48:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:08.117 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf84ca6700 10 monclient: _renew_subs 2023-09-30T00:48:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf84ca6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf7cd99700 4 set_mon_vals no callback set 2023-09-30T00:48:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf84ca6700 1 -- 172.21.15.40:0/2886220718 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf800a9a00 con 0x7fdf800a5fb0 2023-09-30T00:48:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf84ca6700 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf84ca6700 10 monclient: _renew_subs 2023-09-30T00:48:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf84ca6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:08.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf84ca6700 1 -- 172.21.15.40:0/2886220718 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf80152280 con 0x7fdf800a5fb0 2023-09-30T00:48:08.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.113+0000 7fdf84ca6700 1 librados: init done 2023-09-30T00:48:08.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.114+0000 7fdf84ca6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:08.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.114+0000 7fdf84ca6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:08.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.114+0000 7fdf84ca6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:08.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.114+0000 7fdf84ca6700 1 -- 172.21.15.40:0/2886220718 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf800a9bf0 con 0x7fdf800a5fb0 2023-09-30T00:48:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.441+0000 7fdf6e7fc700 1 -- 172.21.15.40:0/2886220718 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fdf70013680 con 0x7fdf800a5fb0 2023-09-30T00:48:08.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.443+0000 7fdf6e7fc700 1 --2- 172.21.15.40:0/2886220718 >> v2:172.21.15.135:6800/116426 conn(0x7fdf60068890 0x7fdf6006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:08.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.443+0000 7fdf6e7fc700 1 -- 172.21.15.40:0/2886220718 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fdf70084930 con 0x7fdf800a5fb0 2023-09-30T00:48:08.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.443+0000 7fdf6e7fc700 10 client.4377.objecter ms_dispatch 0x7fdf80001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:08.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.443+0000 7fdf6e7fc700 3 client.4377.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:08.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.443+0000 7fdf6e7fc700 3 client.4377.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:08.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.443+0000 7fdf6e7fc700 20 client.4377.objecter dump_active .. 0 homeless 2023-09-30T00:48:08.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.443+0000 7fdf6e7fc700 1 -- 172.21.15.40:0/2886220718 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdf700af640 con 0x7fdf800a5fb0 2023-09-30T00:48:08.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.443+0000 7fdf6e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:08.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.443+0000 7fdf6e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:08.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.443+0000 7fdf7dd9b700 1 --2- 172.21.15.40:0/2886220718 >> v2:172.21.15.135:6800/116426 conn(0x7fdf60068890 0x7fdf6006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:08.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.443+0000 7fdf7dd9b700 10 monclient: get_auth_request con 0x7fdf60068890 auth_method 0 2023-09-30T00:48:08.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.443+0000 7fdf6e7fc700 10 client.4377.objecter ms_handle_connect 0x7fdf60068890 2023-09-30T00:48:08.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.443+0000 7fdf7dd9b700 1 --2- 172.21.15.40:0/2886220718 >> v2:172.21.15.135:6800/116426 conn(0x7fdf60068890 0x7fdf6006ad40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdf8013d540 tx=0x7fdf74016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.694+0000 7fdf84ca6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.694+0000 7fdf84ca6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.694+0000 7fdf84ca6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.694+0000 7fdf84ca6700 1 -- 172.21.15.40:0/2886220718 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdf80064a00 con 0x7fdf800a5fb0 2023-09-30T00:48:08.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.700+0000 7fdf6e7fc700 1 -- 172.21.15.40:0/2886220718 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdf700521d0 con 0x7fdf800a5fb0 2023-09-30T00:48:08.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.700+0000 7fdf6e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:08.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.700+0000 7fdf6e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:08.700 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:08.700 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:08.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 10 librados: watch_flush enter 2023-09-30T00:48:08.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 10 librados: watch_flush exit 2023-09-30T00:48:08.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 20 client.4377.objecter shutdown clearing up homeless session... 2023-09-30T00:48:08.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 10 client.4377.objecter successfully canceled tick 2023-09-30T00:48:08.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 1 -- 172.21.15.40:0/2886220718 >> v2:172.21.15.135:6800/116426 conn(0x7fdf60068890 msgr2=0x7fdf6006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:08.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 1 --2- 172.21.15.40:0/2886220718 >> v2:172.21.15.135:6800/116426 conn(0x7fdf60068890 0x7fdf6006ad40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdf8013d540 tx=0x7fdf74016000 comp rx=0 tx=0).stop 2023-09-30T00:48:08.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 10 monclient: shutdown 2023-09-30T00:48:08.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:08.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 1 -- 172.21.15.40:0/2886220718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf800a5fb0 msgr2=0x7fdf8013d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:08.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 1 --2- 172.21.15.40:0/2886220718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf800a5fb0 0x7fdf8013d5a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000e950 tx=0x7fdf70009250 comp rx=0 tx=0).stop 2023-09-30T00:48:08.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 1 -- 172.21.15.40:0/2886220718 shutdown_connections 2023-09-30T00:48:08.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 1 --2- 172.21.15.40:0/2886220718 >> v2:172.21.15.135:6800/116426 conn(0x7fdf60068890 0x7fdf6006ad40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:08.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 1 --2- 172.21.15.40:0/2886220718 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf800b1460 0x7fdf80151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:08.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 1 --2- 172.21.15.40:0/2886220718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf800a5fb0 0x7fdf8013d5a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:08.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 1 --2- 172.21.15.40:0/2886220718 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf800af410 0x7fdf800a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:08.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 1 -- 172.21.15.40:0/2886220718 >> 172.21.15.40:0/2886220718 conn(0x7fdf800631f0 msgr2=0x7fdf800635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:08.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 1 -- 172.21.15.40:0/2886220718 shutdown_connections 2023-09-30T00:48:08.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.702+0000 7fdf84ca6700 1 -- 172.21.15.40:0/2886220718 wait complete. 2023-09-30T00:48:08.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:08.703+0000 7fdf84ca6700 1 librados: shutdown 2023-09-30T00:48:11.716 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:11.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.856+0000 7f7ac83f0700 10 monclient: get_monmap_and_config 2023-09-30T00:48:11.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.856+0000 7f7ac83f0700 10 monclient: build_initial_monmap 2023-09-30T00:48:11.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.856+0000 7f7ac83f0700 10 monclient: monmap: 2023-09-30T00:48:11.857 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:11.857 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:11.857 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:11.857223+0000 2023-09-30T00:48:11.857 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:11.857223+0000 2023-09-30T00:48:11.857 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:11.857 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:11.857 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:11.857 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:11.857 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:11.857 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:11.858 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:11.858 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:11.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.856+0000 7f7ac83f0700 1 Processor -- start 2023-09-30T00:48:11.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.856+0000 7f7ac83f0700 1 -- start start 2023-09-30T00:48:11.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.856+0000 7f7ac83f0700 10 monclient: init 2023-09-30T00:48:11.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.857+0000 7f7ac83f0700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:11.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.857+0000 7f7ac83f0700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2023-09-30T00:48:11.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.857+0000 7f7ac83f0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ac00b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:11.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.857+0000 7f7ac83f0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7ac00b1830 0x7f7ac00a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:11.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.857+0000 7f7ac83f0700 10 monclient(hunting): picked mon.noname-a con 0x7f7ac00b1830 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:11.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.857+0000 7f7ac83f0700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7ac013ac20 0x7f7ac013b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:11.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.857+0000 7f7ac83f0700 10 monclient(hunting): picked mon.noname-c con 0x7f7ac013ac20 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:11.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.857+0000 7f7ac83f0700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac00a5b20 con 0x7f7ac00b1460 2023-09-30T00:48:11.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.857+0000 7f7ac83f0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:11.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.857+0000 7f7ac83f0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac0141e40 con 0x7f7ac00b1830 2023-09-30T00:48:11.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.857+0000 7f7ac83f0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:11.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.857+0000 7f7ac83f0700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac00a64f0 con 0x7f7ac013ac20 2023-09-30T00:48:11.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.857+0000 7f7ac83f0700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:11.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.857+0000 7f7ac83f0700 10 monclient(hunting): authenticate will time out at 5353.158758s 2023-09-30T00:48:11.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.858+0000 7f7ac598b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7ac00b1830 0x7f7ac00a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:11.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.858+0000 7f7ac598b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7ac00b1830 0x7f7ac00a5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33976/0 (socket says 172.21.15.40:33976) 2023-09-30T00:48:11.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.858+0000 7f7ac598b700 1 -- 172.21.15.40:0/1535476872 learned_addr learned my addr 172.21.15.40:0/1535476872 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:11.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.858+0000 7f7ac598b700 10 monclient(hunting): get_auth_request con 0x7f7ac00b1830 auth_method 0 2023-09-30T00:48:11.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.858+0000 7f7ac598b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:11.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.858+0000 7f7ac598b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:11.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.858+0000 7f7ac598b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:11.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.858+0000 7f7ac698d700 1 --2- 172.21.15.40:0/1535476872 >> v2:172.21.15.135:3300/0 conn(0x7f7ac013ac20 0x7f7ac013b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:11.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.859+0000 7f7ac698d700 10 monclient(hunting): get_auth_request con 0x7f7ac013ac20 auth_method 0 2023-09-30T00:48:11.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.859+0000 7f7ac698d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:11.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.859+0000 7f7ac698d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:11.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.859+0000 7f7ac698d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:11.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.859+0000 7f7ac598b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:11.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.859+0000 7f7ac598b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:11.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.859+0000 7f7ac598b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:11.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.861+0000 7f7ac698d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:11.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.861+0000 7f7ac698d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:11.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.861+0000 7f7ac698d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:11.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.862+0000 7f7ac598b700 10 monclient(hunting): handle_auth_done global_id 4380 payload 274 2023-09-30T00:48:11.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.862+0000 7f7ac598b700 1 -- 172.21.15.40:0/1535476872 >> v2:172.21.15.135:3300/0 conn(0x7f7ac013ac20 msgr2=0x7f7ac013b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:11.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.862+0000 7f7ac598b700 1 --2- 172.21.15.40:0/1535476872 >> v2:172.21.15.135:3300/0 conn(0x7f7ac013ac20 0x7f7ac013b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:11.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.862+0000 7f7ac598b700 1 -- 172.21.15.40:0/1535476872 >> v1:172.21.15.135:6789/0 conn(0x7f7ac00b1460 legacy=0x7f7ac0141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:11.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.862+0000 7f7ac598b700 10 monclient: _finish_hunting 0 2023-09-30T00:48:11.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.862+0000 7f7ac598b700 1 monclient: found mon.noname-a 2023-09-30T00:48:11.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.862+0000 7f7ac598b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:11.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.862+0000 7f7ac598b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:11.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.862+0000 7f7ac598b700 1 -- 172.21.15.40:0/1535476872 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac00a71f0 con 0x7f7ac00b1830 2023-09-30T00:48:11.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.864+0000 7f7ac598b700 1 --2- 172.21.15.40:0/1535476872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ac00b1830 0x7f7ac00a5fb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00099e0 tx=0x7f7ab0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=778e529ce36bfb43 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.867+0000 7f7ac518a700 1 -- 172.21.15.40:0/1535476872 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab000b7e0 con 0x7f7ac00b1830 2023-09-30T00:48:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.867+0000 7f7ac518a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.867+0000 7f7ac518a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.867+0000 7f7ac518a700 10 monclient: dump: 2023-09-30T00:48:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:11.869 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.867+0000 7f7ac518a700 10 monclient: _finish_auth 0 2023-09-30T00:48:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.867+0000 7f7ac518a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.867+0000 7f7ac83f0700 5 monclient: authenticate success, global_id 4380 2023-09-30T00:48:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.867+0000 7f7ac83f0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.869+0000 7f7ac518a700 1 -- 172.21.15.40:0/1535476872 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab000b940 con 0x7f7ac00b1830 2023-09-30T00:48:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.869+0000 7f7ac518a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.869+0000 7f7ac518a700 1 -- 172.21.15.40:0/1535476872 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab000bce0 con 0x7f7ac00b1830 2023-09-30T00:48:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.869+0000 7f7ac518a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.869+0000 7f7ac518a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.869+0000 7f7ac518a700 10 monclient: dump: 2023-09-30T00:48:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:11.871 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.869+0000 7f7ac83f0700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.869+0000 7f7ac83f0700 4 set_mon_vals no callback set 2023-09-30T00:48:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.870+0000 7f7ac83f0700 10 monclient: shutdown 2023-09-30T00:48:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.870+0000 7f7ac83f0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.870+0000 7f7ac83f0700 1 -- 172.21.15.40:0/1535476872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ac00b1830 msgr2=0x7f7ac00a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.870+0000 7f7ac83f0700 1 --2- 172.21.15.40:0/1535476872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ac00b1830 0x7f7ac00a5fb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00099e0 tx=0x7f7ab0009250 comp rx=0 tx=0).stop 2023-09-30T00:48:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.870+0000 7f7ac83f0700 1 -- 172.21.15.40:0/1535476872 shutdown_connections 2023-09-30T00:48:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.870+0000 7f7ac83f0700 1 --2- 172.21.15.40:0/1535476872 >> v2:172.21.15.135:3300/0 conn(0x7f7ac013ac20 0x7f7ac013b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.870+0000 7f7ac83f0700 1 --2- 172.21.15.40:0/1535476872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ac00b1830 0x7f7ac00a5fb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.870+0000 7f7ac83f0700 1 -- 172.21.15.40:0/1535476872 >> 172.21.15.40:0/1535476872 conn(0x7f7ac00631f0 msgr2=0x7f7ac00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.870+0000 7f7ac83f0700 1 -- 172.21.15.40:0/1535476872 shutdown_connections 2023-09-30T00:48:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.870+0000 7f7ac83f0700 1 -- 172.21.15.40:0/1535476872 wait complete. 2023-09-30T00:48:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.871+0000 7f7ac83f0700 10 monclient: build_initial_monmap 2023-09-30T00:48:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.871+0000 7f7ac83f0700 10 monclient: monmap: 2023-09-30T00:48:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:11.875 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.871+0000 7f7ac83f0700 1 librados: starting msgr at 2023-09-30T00:48:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.871+0000 7f7ac83f0700 1 librados: starting objecter 2023-09-30T00:48:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.871+0000 7f7ac83f0700 1 Processor -- start 2023-09-30T00:48:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.871+0000 7f7ac83f0700 1 -- start start 2023-09-30T00:48:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.871+0000 7f7ac83f0700 1 librados: setting wanted keys 2023-09-30T00:48:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.871+0000 7f7ac83f0700 1 librados: calling monclient init 2023-09-30T00:48:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.871+0000 7f7ac83f0700 10 monclient: init 2023-09-30T00:48:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:48:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ac00b1460 0x7f7ac00a7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 10 monclient(hunting): picked mon.noname-a con 0x7f7ac00b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ac00b1830 0x7f7ac013dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 10 monclient(hunting): picked mon.noname-b con 0x7f7ac00b1830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ac013ac20 0x7f7ac00a9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 10 monclient(hunting): picked mon.noname-c con 0x7f7ac013ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac00a9550 con 0x7f7ac00b1460 2023-09-30T00:48:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac00ab670 con 0x7f7ac00b1830 2023-09-30T00:48:11.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:11.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac00a9770 con 0x7f7ac013ac20 2023-09-30T00:48:11.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:11.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.872+0000 7f7ac83f0700 10 monclient(hunting): authenticate will time out at 5353.173559s 2023-09-30T00:48:11.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.873+0000 7f7ac618c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ac00b1460 0x7f7ac00a7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:11.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.873+0000 7f7ac698d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ac013ac20 0x7f7ac00a9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:11.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.873+0000 7f7ac618c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ac00b1460 0x7f7ac00a7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33988/0 (socket says 172.21.15.40:33988) 2023-09-30T00:48:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.873+0000 7f7ac618c700 1 -- 172.21.15.40:0/2173483842 learned_addr learned my addr 172.21.15.40:0/2173483842 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.873+0000 7f7ac618c700 10 monclient(hunting): get_auth_request con 0x7f7ac00b1460 auth_method 0 2023-09-30T00:48:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.873+0000 7f7ac618c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.873+0000 7f7ac618c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.873+0000 7f7ac618c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.873+0000 7f7ac698d700 10 monclient(hunting): get_auth_request con 0x7f7ac013ac20 auth_method 0 2023-09-30T00:48:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.873+0000 7f7ac698d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.873+0000 7f7ac698d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.873+0000 7f7ac698d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.873+0000 7f7ac598b700 1 --2- 172.21.15.40:0/2173483842 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ac00b1830 0x7f7ac013dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.874+0000 7f7ac618c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.874+0000 7f7ac618c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.874+0000 7f7ac618c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.874+0000 7f7ac698d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.874+0000 7f7ac698d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.874+0000 7f7ac698d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.874+0000 7f7ac598b700 10 monclient(hunting): get_auth_request con 0x7f7ac00b1830 auth_method 0 2023-09-30T00:48:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.874+0000 7f7ac598b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.874+0000 7f7ac598b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.874+0000 7f7ac598b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.875+0000 7f7ac598b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.875+0000 7f7ac598b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.875+0000 7f7ac598b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.878+0000 7f7ac618c700 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2023-09-30T00:48:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.878+0000 7f7ac618c700 1 -- 172.21.15.40:0/2173483842 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ac013ac20 msgr2=0x7f7ac00a9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.878+0000 7f7ac618c700 1 --2- 172.21.15.40:0/2173483842 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ac013ac20 0x7f7ac00a9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.878+0000 7f7ac618c700 1 -- 172.21.15.40:0/2173483842 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ac00b1830 msgr2=0x7f7ac013dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.878+0000 7f7ac618c700 1 --2- 172.21.15.40:0/2173483842 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ac00b1830 0x7f7ac013dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.878+0000 7f7ac618c700 10 monclient: _finish_hunting 0 2023-09-30T00:48:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.878+0000 7f7ac618c700 1 monclient: found mon.noname-a 2023-09-30T00:48:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.878+0000 7f7ac618c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.878+0000 7f7ac618c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.878+0000 7f7ac618c700 1 -- 172.21.15.40:0/2173483842 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab00094f0 con 0x7f7ac00b1460 2023-09-30T00:48:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.880+0000 7f7ac618c700 1 --2- 172.21.15.40:0/2173483842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ac00b1460 0x7f7ac00a7980 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7abc00c590 tx=0x7f7abc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.880+0000 7f7ab67fc700 10 client.?.objecter ms_handle_connect 0x7f7ac00b1460 2023-09-30T00:48:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.880+0000 7f7ab67fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ab67fc700 1 -- 172.21.15.40:0/2173483842 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc011590 con 0x7f7ac00b1460 2023-09-30T00:48:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ab67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ab67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ab67fc700 10 monclient: dump: 2023-09-30T00:48:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:11.887 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ab67fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ab67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ab67fc700 1 -- 172.21.15.40:0/2173483842 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc011710 con 0x7f7ac00b1460 2023-09-30T00:48:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ac83f0700 5 monclient: authenticate success, global_id 4383 2023-09-30T00:48:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ac83f0700 10 monclient: _renew_subs 2023-09-30T00:48:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ac83f0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ac83f0700 1 -- 172.21.15.40:0/2173483842 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac013ec70 con 0x7f7ac00b1460 2023-09-30T00:48:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ab67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ab67fc700 1 -- 172.21.15.40:0/2173483842 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc011ad0 con 0x7f7ac00b1460 2023-09-30T00:48:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ab67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ab67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ab67fc700 10 monclient: dump: 2023-09-30T00:48:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:11.889 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ac4989700 4 set_mon_vals no callback set 2023-09-30T00:48:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ac83f0700 10 client.4383.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ac83f0700 10 monclient: _renew_subs 2023-09-30T00:48:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ac83f0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ac83f0700 1 -- 172.21.15.40:0/2173483842 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac00a9ef0 con 0x7f7ac00b1460 2023-09-30T00:48:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.885+0000 7f7ac83f0700 1 librados: init done 2023-09-30T00:48:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.887+0000 7f7ac83f0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.887+0000 7f7ac83f0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.887+0000 7f7ac83f0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:11.887+0000 7f7ac83f0700 1 -- 172.21.15.40:0/2173483842 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac00a99d0 con 0x7f7ac00b1460 2023-09-30T00:48:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.193+0000 7f7ab67fc700 1 -- 172.21.15.40:0/2173483842 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7abc01b3b0 con 0x7f7ac00b1460 2023-09-30T00:48:12.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.194+0000 7f7ab67fc700 1 --2- 172.21.15.40:0/2173483842 >> v2:172.21.15.135:6800/116426 conn(0x7f7aa4068890 0x7f7aa406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:12.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.194+0000 7f7ab67fc700 1 -- 172.21.15.40:0/2173483842 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7abc0835c0 con 0x7f7ac00b1460 2023-09-30T00:48:12.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.194+0000 7f7ab67fc700 10 client.4383.objecter ms_dispatch 0x7f7ac0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:12.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.194+0000 7f7ab67fc700 3 client.4383.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:12.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.194+0000 7f7ab67fc700 3 client.4383.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:12.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.194+0000 7f7ab67fc700 20 client.4383.objecter dump_active .. 0 homeless 2023-09-30T00:48:12.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.194+0000 7f7ac598b700 1 --2- 172.21.15.40:0/2173483842 >> v2:172.21.15.135:6800/116426 conn(0x7f7aa4068890 0x7f7aa406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:12.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.194+0000 7f7ac598b700 10 monclient: get_auth_request con 0x7f7aa4068890 auth_method 0 2023-09-30T00:48:12.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.195+0000 7f7ac598b700 1 --2- 172.21.15.40:0/2173483842 >> v2:172.21.15.135:6800/116426 conn(0x7f7aa4068890 0x7f7aa406ad40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00099e0 tx=0x7f7ab0017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:12.196 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.195+0000 7f7ab67fc700 10 client.4383.objecter ms_handle_connect 0x7f7aa4068890 2023-09-30T00:48:12.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.207+0000 7f7ab67fc700 1 -- 172.21.15.40:0/2173483842 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7abc050e60 con 0x7f7ac00b1460 2023-09-30T00:48:12.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.207+0000 7f7ab67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:12.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.207+0000 7f7ab67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:12.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.515+0000 7f7ac83f0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:12.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.515+0000 7f7ac83f0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:12.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.515+0000 7f7ac83f0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:48:12.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.515+0000 7f7ac83f0700 1 -- 172.21.15.40:0/2173483842 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7ac00a99d0 con 0x7f7ac00b1460 2023-09-30T00:48:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.520+0000 7f7ab67fc700 1 -- 172.21.15.40:0/2173483842 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7abc017080 con 0x7f7ac00b1460 2023-09-30T00:48:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.520+0000 7f7ab67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.520+0000 7f7ab67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:12.520 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:12.521 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:12.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 10 librados: watch_flush enter 2023-09-30T00:48:12.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 10 librados: watch_flush exit 2023-09-30T00:48:12.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 20 client.4383.objecter shutdown clearing up homeless session... 2023-09-30T00:48:12.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 10 client.4383.objecter successfully canceled tick 2023-09-30T00:48:12.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 1 -- 172.21.15.40:0/2173483842 >> v2:172.21.15.135:6800/116426 conn(0x7f7aa4068890 msgr2=0x7f7aa406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:12.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 1 --2- 172.21.15.40:0/2173483842 >> v2:172.21.15.135:6800/116426 conn(0x7f7aa4068890 0x7f7aa406ad40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7ab00099e0 tx=0x7f7ab0017000 comp rx=0 tx=0).stop 2023-09-30T00:48:12.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 10 monclient: shutdown 2023-09-30T00:48:12.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:12.523 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 1 -- 172.21.15.40:0/2173483842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ac00b1460 msgr2=0x7f7ac00a7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 1 --2- 172.21.15.40:0/2173483842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ac00b1460 0x7f7ac00a7980 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7abc00c590 tx=0x7f7abc00c8a0 comp rx=0 tx=0).stop 2023-09-30T00:48:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 1 -- 172.21.15.40:0/2173483842 shutdown_connections 2023-09-30T00:48:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 1 --2- 172.21.15.40:0/2173483842 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7ac013ac20 0x7f7ac00a9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 1 --2- 172.21.15.40:0/2173483842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ac00b1460 0x7f7ac00a7980 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 1 --2- 172.21.15.40:0/2173483842 >> v2:172.21.15.135:6800/116426 conn(0x7f7aa4068890 0x7f7aa406ad40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 1 --2- 172.21.15.40:0/2173483842 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7ac00b1830 0x7f7ac013dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 1 -- 172.21.15.40:0/2173483842 >> 172.21.15.40:0/2173483842 conn(0x7f7ac00631f0 msgr2=0x7f7ac00635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 1 -- 172.21.15.40:0/2173483842 shutdown_connections 2023-09-30T00:48:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 1 -- 172.21.15.40:0/2173483842 wait complete. 2023-09-30T00:48:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:12.522+0000 7f7ac83f0700 1 librados: shutdown 2023-09-30T00:48:15.539 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:15.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.678+0000 7fdac857f700 10 monclient: get_monmap_and_config 2023-09-30T00:48:15.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.678+0000 7fdac857f700 10 monclient: build_initial_monmap 2023-09-30T00:48:15.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.678+0000 7fdac857f700 10 monclient: monmap: 2023-09-30T00:48:15.679 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:15.679 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:15.679 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:15.679235+0000 2023-09-30T00:48:15.679 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:15.679235+0000 2023-09-30T00:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.678+0000 7fdac857f700 1 Processor -- start 2023-09-30T00:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.678+0000 7fdac857f700 1 -- start start 2023-09-30T00:48:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.678+0000 7fdac857f700 10 monclient: init 2023-09-30T00:48:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.679+0000 7fdac857f700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.679+0000 7fdac857f700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-09-30T00:48:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.679+0000 7fdac857f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdac00b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.679+0000 7fdac857f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdac00b1830 addr v1:172.21.15.40:6789/0 2023-09-30T00:48:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.679+0000 7fdac857f700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdac00a5fb0 0x7fdac013b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.679+0000 7fdac857f700 10 monclient(hunting): picked mon.noname-c con 0x7fdac00a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.679+0000 7fdac857f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac00a5b20 con 0x7fdac00b1830 2023-09-30T00:48:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.679+0000 7fdac857f700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac00a71f0 con 0x7fdac00b1460 2023-09-30T00:48:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.679+0000 7fdac857f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.679+0000 7fdac857f700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdac0141e40 con 0x7fdac00a5fb0 2023-09-30T00:48:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.679+0000 7fdac857f700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.679+0000 7fdac857f700 10 monclient(hunting): authenticate will time out at 5356.980722s 2023-09-30T00:48:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.680+0000 7fdac5b1a700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdac00b1830 0x7fdac013ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:40358/0 (socket says 172.21.15.40:40358) 2023-09-30T00:48:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.680+0000 7fdac5b1a700 1 -- 172.21.15.40:0/2070163377 learned_addr learned my addr 172.21.15.40:0/2070163377 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:48:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.680+0000 7fdac6b1c700 1 --2- 172.21.15.40:0/2070163377 >> v2:172.21.15.135:3300/0 conn(0x7fdac00a5fb0 0x7fdac013b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.681+0000 7fdac6b1c700 10 monclient(hunting): get_auth_request con 0x7fdac00a5fb0 auth_method 0 2023-09-30T00:48:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.681+0000 7fdac6b1c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.681+0000 7fdac6b1c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.681+0000 7fdac6b1c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:15.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.682+0000 7fdac6b1c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:15.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.682+0000 7fdac6b1c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:15.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.682+0000 7fdac6b1c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:15.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.683+0000 7fdac5319700 1 -- 172.21.15.40:0/2070163377 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902435203 0 0) 0x7fdac00a5b20 con 0x7fdac00b1830 2023-09-30T00:48:15.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.683+0000 7fdac5319700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.683+0000 7fdac5319700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.683+0000 7fdac5319700 10 monclient(hunting): my global_id is 4386 2023-09-30T00:48:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.683+0000 7fdac5319700 1 -- 172.21.15.40:0/2070163377 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaac003570 con 0x7fdac00b1830 2023-09-30T00:48:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.685+0000 7fdac6b1c700 10 monclient(hunting): handle_auth_done global_id 4417 payload 274 2023-09-30T00:48:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.685+0000 7fdac6b1c700 1 -- 172.21.15.40:0/2070163377 >> v1:172.21.15.135:6789/0 conn(0x7fdac00b1460 legacy=0x7fdac0141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.685+0000 7fdac6b1c700 1 -- 172.21.15.40:0/2070163377 >> v1:172.21.15.40:6789/0 conn(0x7fdac00b1830 legacy=0x7fdac013ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.685+0000 7fdac6b1c700 10 monclient: _finish_hunting 0 2023-09-30T00:48:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.685+0000 7fdac6b1c700 1 monclient: found mon.noname-c 2023-09-30T00:48:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.685+0000 7fdac6b1c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.685+0000 7fdac6b1c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:48:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.685+0000 7fdac6b1c700 1 -- 172.21.15.40:0/2070163377 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac00a74d0 con 0x7fdac00a5fb0 2023-09-30T00:48:15.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.689+0000 7fdac6b1c700 1 --2- 172.21.15.40:0/2070163377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdac00a5fb0 0x7fdac013b330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009e90 tx=0x7fdab8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f3bd0a7031223bd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.695+0000 7fdac5319700 1 -- 172.21.15.40:0/2070163377 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab800e070 con 0x7fdac00a5fb0 2023-09-30T00:48:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.695+0000 7fdac5319700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.695+0000 7fdac5319700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.695+0000 7fdac5319700 10 monclient: dump: 2023-09-30T00:48:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:15.697 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.695+0000 7fdac5319700 10 monclient: _finish_auth 0 2023-09-30T00:48:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.695+0000 7fdac5319700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.695+0000 7fdac5319700 1 -- 172.21.15.40:0/2070163377 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab800bc40 con 0x7fdac00a5fb0 2023-09-30T00:48:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.695+0000 7fdac5319700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.695+0000 7fdac5319700 1 -- 172.21.15.40:0/2070163377 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab8012430 con 0x7fdac00a5fb0 2023-09-30T00:48:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.695+0000 7fdac5319700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.695+0000 7fdac5319700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.695+0000 7fdac5319700 10 monclient: dump: 2023-09-30T00:48:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:15.699 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:15.699 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:15.699 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:15.699 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:15.699 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:15.699 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:15.699 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:15.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.695+0000 7fdac857f700 5 monclient: authenticate success, global_id 4417 2023-09-30T00:48:15.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.695+0000 7fdac857f700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:15.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.696+0000 7fdac857f700 4 set_mon_vals no callback set 2023-09-30T00:48:15.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.696+0000 7fdac857f700 10 monclient: shutdown 2023-09-30T00:48:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.696+0000 7fdac857f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.696+0000 7fdac857f700 1 -- 172.21.15.40:0/2070163377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdac00a5fb0 msgr2=0x7fdac013b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.696+0000 7fdac857f700 1 --2- 172.21.15.40:0/2070163377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdac00a5fb0 0x7fdac013b330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdab8009e90 tx=0x7fdab8009780 comp rx=0 tx=0).stop 2023-09-30T00:48:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.696+0000 7fdac857f700 1 -- 172.21.15.40:0/2070163377 shutdown_connections 2023-09-30T00:48:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.696+0000 7fdac857f700 1 --2- 172.21.15.40:0/2070163377 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdac00a5fb0 0x7fdac013b330 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.696+0000 7fdac857f700 1 -- 172.21.15.40:0/2070163377 >> 172.21.15.40:0/2070163377 conn(0x7fdac00631f0 msgr2=0x7fdac00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.696+0000 7fdac857f700 1 -- 172.21.15.40:0/2070163377 shutdown_connections 2023-09-30T00:48:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.696+0000 7fdac857f700 1 -- 172.21.15.40:0/2070163377 wait complete. 2023-09-30T00:48:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.697+0000 7fdac857f700 10 monclient: build_initial_monmap 2023-09-30T00:48:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.697+0000 7fdac857f700 10 monclient: monmap: 2023-09-30T00:48:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:15.704 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.697+0000 7fdac857f700 1 librados: starting msgr at 2023-09-30T00:48:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.697+0000 7fdac857f700 1 librados: starting objecter 2023-09-30T00:48:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.697+0000 7fdac857f700 1 Processor -- start 2023-09-30T00:48:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.697+0000 7fdac857f700 1 -- start start 2023-09-30T00:48:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.697+0000 7fdac857f700 1 librados: setting wanted keys 2023-09-30T00:48:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.697+0000 7fdac857f700 1 librados: calling monclient init 2023-09-30T00:48:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.697+0000 7fdac857f700 10 monclient: init 2023-09-30T00:48:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:48:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdac00a5fb0 0x7fdac00a9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 10 monclient(hunting): picked mon.noname-a con 0x7fdac00a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdac00b1460 0x7fdac013e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 10 monclient(hunting): picked mon.noname-b con 0x7fdac00b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdac00b1830 0x7fdac00aa060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 10 monclient(hunting): picked mon.noname-c con 0x7fdac00b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac00a9940 con 0x7fdac00a5fb0 2023-09-30T00:48:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac00ab8e0 con 0x7fdac00b1460 2023-09-30T00:48:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac00a9c50 con 0x7fdac00b1830 2023-09-30T00:48:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.698+0000 7fdac857f700 10 monclient(hunting): authenticate will time out at 5356.999514s 2023-09-30T00:48:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac5b1a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdac00b1460 0x7fdac013e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac631b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdac00a5fb0 0x7fdac00a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac6b1c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdac00b1830 0x7fdac00aa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac5b1a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdac00b1460 0x7fdac013e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:51588/0 (socket says 172.21.15.40:51588) 2023-09-30T00:48:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac5b1a700 1 -- 172.21.15.40:0/2497060701 learned_addr learned my addr 172.21.15.40:0/2497060701 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac5b1a700 10 monclient(hunting): get_auth_request con 0x7fdac00b1460 auth_method 0 2023-09-30T00:48:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac5b1a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac5b1a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac5b1a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac631b700 10 monclient(hunting): get_auth_request con 0x7fdac00a5fb0 auth_method 0 2023-09-30T00:48:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac631b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac631b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac631b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac6b1c700 10 monclient(hunting): get_auth_request con 0x7fdac00b1830 auth_method 0 2023-09-30T00:48:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac6b1c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac6b1c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.699+0000 7fdac6b1c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.700+0000 7fdac5b1a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.700+0000 7fdac5b1a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.700+0000 7fdac5b1a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.700+0000 7fdac631b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.700+0000 7fdac631b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.700+0000 7fdac631b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.700+0000 7fdac6b1c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.700+0000 7fdac6b1c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.700+0000 7fdac6b1c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:15.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.703+0000 7fdac5b1a700 10 monclient(hunting): handle_auth_done global_id 4423 payload 824 2023-09-30T00:48:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.703+0000 7fdac5b1a700 1 -- 172.21.15.40:0/2497060701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdac00b1830 msgr2=0x7fdac00aa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.703+0000 7fdac5b1a700 1 --2- 172.21.15.40:0/2497060701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdac00b1830 0x7fdac00aa060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.703+0000 7fdac5b1a700 1 -- 172.21.15.40:0/2497060701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdac00a5fb0 msgr2=0x7fdac00a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.703+0000 7fdac5b1a700 1 --2- 172.21.15.40:0/2497060701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdac00a5fb0 0x7fdac00a9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.703+0000 7fdac5b1a700 10 monclient: _finish_hunting 0 2023-09-30T00:48:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.703+0000 7fdac5b1a700 1 monclient: found mon.noname-b 2023-09-30T00:48:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.703+0000 7fdac5b1a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.703+0000 7fdac5b1a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.703+0000 7fdac5b1a700 1 -- 172.21.15.40:0/2497060701 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab8009a20 con 0x7fdac00b1460 2023-09-30T00:48:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.705+0000 7fdac5b1a700 1 --2- 172.21.15.40:0/2497060701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdac00b1460 0x7fdac013e2d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdab000c590 tx=0x7fdab0007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.705+0000 7fdab67fc700 10 client.?.objecter ms_handle_connect 0x7fdac00b1460 2023-09-30T00:48:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.705+0000 7fdab67fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdab67fc700 1 -- 172.21.15.40:0/2497060701 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0005b30 con 0x7fdac00b1460 2023-09-30T00:48:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdab67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdab67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdab67fc700 10 monclient: dump: 2023-09-30T00:48:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:15.713 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdab67fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdab67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdab67fc700 1 -- 172.21.15.40:0/2497060701 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab0005c90 con 0x7fdac00b1460 2023-09-30T00:48:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdab67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdab67fc700 1 -- 172.21.15.40:0/2497060701 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab0007320 con 0x7fdac00b1460 2023-09-30T00:48:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdac857f700 5 monclient: authenticate success, global_id 4423 2023-09-30T00:48:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdac857f700 10 monclient: _renew_subs 2023-09-30T00:48:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdac857f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdac857f700 1 -- 172.21.15.40:0/2497060701 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac013f020 con 0x7fdac00b1460 2023-09-30T00:48:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdab67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdac4b18700 4 set_mon_vals no callback set 2023-09-30T00:48:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdab67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.709+0000 7fdab67fc700 10 monclient: dump: 2023-09-30T00:48:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:15.715 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.710+0000 7fdac857f700 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.710+0000 7fdac857f700 10 monclient: _renew_subs 2023-09-30T00:48:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.710+0000 7fdac857f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.710+0000 7fdac857f700 1 -- 172.21.15.40:0/2497060701 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac00a79f0 con 0x7fdac00b1460 2023-09-30T00:48:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.710+0000 7fdac857f700 1 librados: init done 2023-09-30T00:48:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.711+0000 7fdac857f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.711+0000 7fdac857f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.711+0000 7fdac857f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.711+0000 7fdac857f700 1 -- 172.21.15.40:0/2497060701 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac00abc10 con 0x7fdac00b1460 2023-09-30T00:48:15.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.961+0000 7fdab67fc700 1 -- 172.21.15.40:0/2497060701 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fdab001b3b0 con 0x7fdac00b1460 2023-09-30T00:48:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.962+0000 7fdab67fc700 1 --2- 172.21.15.40:0/2497060701 >> v2:172.21.15.135:6800/116426 conn(0x7fdaa4068960 0x7fdaa406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.962+0000 7fdab67fc700 1 -- 172.21.15.40:0/2497060701 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fdab0083580 con 0x7fdac00b1460 2023-09-30T00:48:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.962+0000 7fdab67fc700 10 client.4423.objecter ms_dispatch 0x7fdac0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.962+0000 7fdab67fc700 3 client.4423.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.962+0000 7fdab67fc700 3 client.4423.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.962+0000 7fdac631b700 1 --2- 172.21.15.40:0/2497060701 >> v2:172.21.15.135:6800/116426 conn(0x7fdaa4068960 0x7fdaa406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.962+0000 7fdab67fc700 20 client.4423.objecter dump_active .. 0 homeless 2023-09-30T00:48:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.962+0000 7fdac631b700 10 monclient: get_auth_request con 0x7fdaa4068960 auth_method 0 2023-09-30T00:48:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.963+0000 7fdab67fc700 10 client.4423.objecter ms_handle_connect 0x7fdaa4068960 2023-09-30T00:48:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.963+0000 7fdac631b700 1 --2- 172.21.15.40:0/2497060701 >> v2:172.21.15.135:6800/116426 conn(0x7fdaa4068960 0x7fdaa406ae10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdabc003a80 tx=0x7fdabc007ec0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.972+0000 7fdab67fc700 1 -- 172.21.15.40:0/2497060701 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdab0050ea0 con 0x7fdac00b1460 2023-09-30T00:48:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.972+0000 7fdab67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:15.972+0000 7fdab67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:16.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.230+0000 7fdac857f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:16.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.230+0000 7fdac857f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:16.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.230+0000 7fdac857f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:16.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.230+0000 7fdac857f700 1 -- 172.21.15.40:0/2497060701 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdac00abc10 con 0x7fdac00b1460 2023-09-30T00:48:16.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.238+0000 7fdab67fc700 1 -- 172.21.15.40:0/2497060701 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdab0017100 con 0x7fdac00b1460 2023-09-30T00:48:16.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.238+0000 7fdab67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:16.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.238+0000 7fdab67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:16.239 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:16.239 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:16.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.240+0000 7fdac857f700 10 librados: watch_flush enter 2023-09-30T00:48:16.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 10 librados: watch_flush exit 2023-09-30T00:48:16.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 20 client.4423.objecter shutdown clearing up homeless session... 2023-09-30T00:48:16.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 10 client.4423.objecter successfully canceled tick 2023-09-30T00:48:16.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 1 -- 172.21.15.40:0/2497060701 >> v2:172.21.15.135:6800/116426 conn(0x7fdaa4068960 msgr2=0x7fdaa406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:16.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 1 --2- 172.21.15.40:0/2497060701 >> v2:172.21.15.135:6800/116426 conn(0x7fdaa4068960 0x7fdaa406ae10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdabc003a80 tx=0x7fdabc007ec0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 10 monclient: shutdown 2023-09-30T00:48:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 1 -- 172.21.15.40:0/2497060701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdac00b1460 msgr2=0x7fdac013e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 1 --2- 172.21.15.40:0/2497060701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdac00b1460 0x7fdac013e2d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdab000c590 tx=0x7fdab0007e00 comp rx=0 tx=0).stop 2023-09-30T00:48:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 1 -- 172.21.15.40:0/2497060701 shutdown_connections 2023-09-30T00:48:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 1 --2- 172.21.15.40:0/2497060701 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdac00b1830 0x7fdac00aa060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 1 --2- 172.21.15.40:0/2497060701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdac00a5fb0 0x7fdac00a9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 1 --2- 172.21.15.40:0/2497060701 >> v2:172.21.15.135:6800/116426 conn(0x7fdaa4068960 0x7fdaa406ae10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 1 --2- 172.21.15.40:0/2497060701 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdac00b1460 0x7fdac013e2d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 1 -- 172.21.15.40:0/2497060701 >> 172.21.15.40:0/2497060701 conn(0x7fdac00631f0 msgr2=0x7fdac00b3500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 1 -- 172.21.15.40:0/2497060701 shutdown_connections 2023-09-30T00:48:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 1 -- 172.21.15.40:0/2497060701 wait complete. 2023-09-30T00:48:16.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:16.241+0000 7fdac857f700 1 librados: shutdown 2023-09-30T00:48:19.256 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:19.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.396+0000 7f747d0a8700 10 monclient: get_monmap_and_config 2023-09-30T00:48:19.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.396+0000 7f747d0a8700 10 monclient: build_initial_monmap 2023-09-30T00:48:19.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.396+0000 7f747d0a8700 10 monclient: monmap: 2023-09-30T00:48:19.397 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:19.397 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:19.397 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:19.397430+0000 2023-09-30T00:48:19.397 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:19.397430+0000 2023-09-30T00:48:19.397 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:19.397 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:19.397 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:19.397 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:19.397 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:19.397 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:19.398 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:19.398 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:19.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 1 Processor -- start 2023-09-30T00:48:19.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 1 -- start start 2023-09-30T00:48:19.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 10 monclient: init 2023-09-30T00:48:19.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:19.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-09-30T00:48:19.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f74780b1460 0x7f74780b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:19.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 10 monclient(hunting): picked mon.noname-c con 0x7f74780b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:19.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74780a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:19.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f747813ac20 0x7f747813b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:19.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 10 monclient(hunting): picked mon.noname-b con 0x7f747813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:19.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74780a5b20 con 0x7f74780a5fb0 2023-09-30T00:48:19.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:19.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74780a71f0 con 0x7f74780b1460 2023-09-30T00:48:19.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:19.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74780a74d0 con 0x7f747813ac20 2023-09-30T00:48:19.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:19.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.397+0000 7f747d0a8700 10 monclient(hunting): authenticate will time out at 5360.698771s 2023-09-30T00:48:19.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.398+0000 7f747759e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f747813ac20 0x7f747813b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:19.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.398+0000 7f7476d9d700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f74780b1460 0x7f74780b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:19.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.398+0000 7f747659c700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f74780a5fb0 0x7f7478141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:58920/0 (socket says 172.21.15.40:58920) 2023-09-30T00:48:19.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.398+0000 7f747659c700 1 -- 172.21.15.40:0/2540130477 learned_addr learned my addr 172.21.15.40:0/2540130477 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:48:19.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.398+0000 7f747759e700 10 monclient(hunting): get_auth_request con 0x7f747813ac20 auth_method 0 2023-09-30T00:48:19.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.398+0000 7f747759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:19.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.398+0000 7f747759e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:19.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.398+0000 7f747759e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:19.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.399+0000 7f7476d9d700 10 monclient(hunting): get_auth_request con 0x7f74780b1460 auth_method 0 2023-09-30T00:48:19.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.399+0000 7f7476d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:19.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.399+0000 7f7476d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:19.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.399+0000 7f7476d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:19.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.399+0000 7f747759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:19.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.399+0000 7f747759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:19.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.399+0000 7f747759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:19.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.399+0000 7f7476d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:19.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.399+0000 7f7476d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:19.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.400+0000 7f7476d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:19.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.401+0000 7f747759e700 10 monclient(hunting): handle_auth_done global_id 4343 payload 274 2023-09-30T00:48:19.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.402+0000 7f747759e700 1 -- 172.21.15.40:0/2540130477 >> v2:172.21.15.135:3300/0 conn(0x7f74780b1460 msgr2=0x7f74780b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:19.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.402+0000 7f747759e700 1 --2- 172.21.15.40:0/2540130477 >> v2:172.21.15.135:3300/0 conn(0x7f74780b1460 0x7f74780b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:19.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.402+0000 7f747759e700 1 -- 172.21.15.40:0/2540130477 >> v1:172.21.15.135:6789/0 conn(0x7f74780a5fb0 legacy=0x7f7478141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:19.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.402+0000 7f747759e700 10 monclient: _finish_hunting 0 2023-09-30T00:48:19.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.402+0000 7f747759e700 1 monclient: found mon.noname-b 2023-09-30T00:48:19.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.402+0000 7f747759e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:19.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.402+0000 7f747759e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:48:19.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.402+0000 7f747759e700 1 -- 172.21.15.40:0/2540130477 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74780a63c0 con 0x7f747813ac20 2023-09-30T00:48:19.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.404+0000 7f747759e700 1 --2- 172.21.15.40:0/2540130477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f747813ac20 0x7f747813b080 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f74700098e0 tx=0x7f74700091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecdde30d03570ddc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:19.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f7475d9b700 1 -- 172.21.15.40:0/2540130477 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747000b640 con 0x7f747813ac20 2023-09-30T00:48:19.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f7475d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:19.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f7475d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:19.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f7475d9b700 10 monclient: dump: 2023-09-30T00:48:19.409 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:19.409 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:19.409 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:19.410 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:19.410 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:19.410 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:19.410 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:19.410 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:19.410 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:19.410 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:19.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f7475d9b700 10 monclient: _finish_auth 0 2023-09-30T00:48:19.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f7475d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:19.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f7475d9b700 1 -- 172.21.15.40:0/2540130477 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747000b7a0 con 0x7f747813ac20 2023-09-30T00:48:19.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f7475d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:19.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f7475d9b700 1 -- 172.21.15.40:0/2540130477 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747000bb40 con 0x7f747813ac20 2023-09-30T00:48:19.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f7475d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:19.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f7475d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:19.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f7475d9b700 10 monclient: dump: 2023-09-30T00:48:19.411 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:19.411 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:19.411 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:19.411 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:19.411 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:19.411 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:19.411 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:19.411 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:19.411 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:19.412 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:19.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f747d0a8700 5 monclient: authenticate success, global_id 4343 2023-09-30T00:48:19.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f747d0a8700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:19.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.408+0000 7f747d0a8700 4 set_mon_vals no callback set 2023-09-30T00:48:19.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.409+0000 7f747d0a8700 10 monclient: shutdown 2023-09-30T00:48:19.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.409+0000 7f747d0a8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:19.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.409+0000 7f747d0a8700 1 -- 172.21.15.40:0/2540130477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f747813ac20 msgr2=0x7f747813b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:19.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.409+0000 7f747d0a8700 1 --2- 172.21.15.40:0/2540130477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f747813ac20 0x7f747813b080 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f74700098e0 tx=0x7f74700091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:19.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.409+0000 7f747d0a8700 1 -- 172.21.15.40:0/2540130477 shutdown_connections 2023-09-30T00:48:19.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.409+0000 7f747d0a8700 1 --2- 172.21.15.40:0/2540130477 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f747813ac20 0x7f747813b080 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:19.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.409+0000 7f747d0a8700 1 --2- 172.21.15.40:0/2540130477 >> v2:172.21.15.135:3300/0 conn(0x7f74780b1460 0x7f74780b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:19.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.409+0000 7f747d0a8700 1 -- 172.21.15.40:0/2540130477 >> 172.21.15.40:0/2540130477 conn(0x7f74780631f0 msgr2=0x7f74780635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:19.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.409+0000 7f747d0a8700 1 -- 172.21.15.40:0/2540130477 shutdown_connections 2023-09-30T00:48:19.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.409+0000 7f747d0a8700 1 -- 172.21.15.40:0/2540130477 wait complete. 2023-09-30T00:48:19.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.410+0000 7f747d0a8700 10 monclient: build_initial_monmap 2023-09-30T00:48:19.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.410+0000 7f747d0a8700 10 monclient: monmap: 2023-09-30T00:48:19.416 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:19.416 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:19.416 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:19.416 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:19.417 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.411+0000 7f747d0a8700 1 librados: starting msgr at 2023-09-30T00:48:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.411+0000 7f747d0a8700 1 librados: starting objecter 2023-09-30T00:48:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.411+0000 7f747d0a8700 1 Processor -- start 2023-09-30T00:48:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.411+0000 7f747d0a8700 1 -- start start 2023-09-30T00:48:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.411+0000 7f747d0a8700 1 librados: setting wanted keys 2023-09-30T00:48:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.411+0000 7f747d0a8700 1 librados: calling monclient init 2023-09-30T00:48:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.411+0000 7f747d0a8700 10 monclient: init 2023-09-30T00:48:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:48:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74780a5fb0 0x7f747813e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 10 monclient(hunting): picked mon.noname-b con 0x7f74780a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74780b1460 0x7f74780a9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 10 monclient(hunting): picked mon.noname-a con 0x7f74780b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f747813ac20 0x7f74780aa180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 10 monclient(hunting): picked mon.noname-c con 0x7f747813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74780a9830 con 0x7f74780b1460 2023-09-30T00:48:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74780ab500 con 0x7f74780a5fb0 2023-09-30T00:48:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74780ab7e0 con 0x7f747813ac20 2023-09-30T00:48:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747d0a8700 10 monclient(hunting): authenticate will time out at 5360.713297s 2023-09-30T00:48:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747759e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f747813ac20 0x7f74780aa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f747659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74780b1460 0x7f74780a9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.412+0000 7f7476d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74780a5fb0 0x7f747813e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f7476d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74780a5fb0 0x7f747813e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:51602/0 (socket says 172.21.15.40:51602) 2023-09-30T00:48:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f7476d9d700 1 -- 172.21.15.40:0/4107135346 learned_addr learned my addr 172.21.15.40:0/4107135346 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f7476d9d700 10 monclient(hunting): get_auth_request con 0x7f74780a5fb0 auth_method 0 2023-09-30T00:48:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f7476d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f7476d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f7476d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f747759e700 10 monclient(hunting): get_auth_request con 0x7f747813ac20 auth_method 0 2023-09-30T00:48:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f747759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f747759e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f747759e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f747659c700 10 monclient(hunting): get_auth_request con 0x7f74780b1460 auth_method 0 2023-09-30T00:48:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f747659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f747659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f747659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f7476d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f7476d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:19.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f7476d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:19.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f747759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:19.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f747759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:19.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.413+0000 7f747759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:19.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.414+0000 7f747659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:19.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.414+0000 7f747659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:19.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.414+0000 7f747659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:19.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.416+0000 7f7476d9d700 10 monclient(hunting): handle_auth_done global_id 4432 payload 824 2023-09-30T00:48:19.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.416+0000 7f7476d9d700 1 -- 172.21.15.40:0/4107135346 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f747813ac20 msgr2=0x7f74780aa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:19.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.416+0000 7f7476d9d700 1 --2- 172.21.15.40:0/4107135346 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f747813ac20 0x7f74780aa180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:19.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.416+0000 7f7476d9d700 1 -- 172.21.15.40:0/4107135346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74780b1460 msgr2=0x7f74780a9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:19.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.416+0000 7f7476d9d700 1 --2- 172.21.15.40:0/4107135346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74780b1460 0x7f74780a9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:19.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.416+0000 7f7476d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:48:19.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.416+0000 7f7476d9d700 1 monclient: found mon.noname-b 2023-09-30T00:48:19.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.416+0000 7f7476d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:19.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.417+0000 7f7476d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:19.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.417+0000 7f7476d9d700 1 -- 172.21.15.40:0/4107135346 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7470009420 con 0x7f74780a5fb0 2023-09-30T00:48:19.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.419+0000 7f7476d9d700 1 --2- 172.21.15.40:0/4107135346 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74780a5fb0 0x7f747813e130 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7468009ef0 tx=0x7f746800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:19.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.419+0000 7f746effd700 10 client.?.objecter ms_handle_connect 0x7f74780a5fb0 2023-09-30T00:48:19.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.419+0000 7f746effd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:19.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f746effd700 1 -- 172.21.15.40:0/4107135346 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746800cb70 con 0x7f74780a5fb0 2023-09-30T00:48:19.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f746effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:19.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f746effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:19.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f746effd700 10 monclient: dump: 2023-09-30T00:48:19.425 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:19.425 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:19.425 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:19.425 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:19.425 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:19.425 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:19.425 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:19.425 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:19.425 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:19.425 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:19.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f746effd700 10 monclient: _finish_auth 0 2023-09-30T00:48:19.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f746effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:19.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f746effd700 1 -- 172.21.15.40:0/4107135346 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746800f040 con 0x7f74780a5fb0 2023-09-30T00:48:19.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f747d0a8700 5 monclient: authenticate success, global_id 4432 2023-09-30T00:48:19.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f747d0a8700 10 monclient: _renew_subs 2023-09-30T00:48:19.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f747d0a8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:19.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f747d0a8700 1 -- 172.21.15.40:0/4107135346 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747813ede0 con 0x7f74780a5fb0 2023-09-30T00:48:19.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f746effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:19.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f746effd700 1 -- 172.21.15.40:0/4107135346 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7468014480 con 0x7f74780a5fb0 2023-09-30T00:48:19.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f746effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:19.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f746effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:19.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f746effd700 10 monclient: dump: 2023-09-30T00:48:19.426 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:19.427 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f747559a700 4 set_mon_vals no callback set 2023-09-30T00:48:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f747d0a8700 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f747d0a8700 10 monclient: _renew_subs 2023-09-30T00:48:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f747d0a8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f747d0a8700 1 -- 172.21.15.40:0/4107135346 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74780a9180 con 0x7f74780a5fb0 2023-09-30T00:48:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.423+0000 7f747d0a8700 1 librados: init done 2023-09-30T00:48:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.424+0000 7f747d0a8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.424+0000 7f747d0a8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.424+0000 7f747d0a8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.424+0000 7f747d0a8700 1 -- 172.21.15.40:0/4107135346 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74780a9970 con 0x7f74780a5fb0 2023-09-30T00:48:19.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.678+0000 7f746effd700 1 -- 172.21.15.40:0/4107135346 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f74680147f0 con 0x7f74780a5fb0 2023-09-30T00:48:19.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.678+0000 7f746effd700 1 --2- 172.21.15.40:0/4107135346 >> v2:172.21.15.135:6800/116426 conn(0x7f7458068940 0x7f745806adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:19.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.679+0000 7f746effd700 1 -- 172.21.15.40:0/4107135346 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7468084f50 con 0x7f74780a5fb0 2023-09-30T00:48:19.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.679+0000 7f746effd700 10 client.4432.objecter ms_dispatch 0x7f7478001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:19.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.679+0000 7f746effd700 3 client.4432.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:19.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.679+0000 7f746effd700 3 client.4432.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:19.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.679+0000 7f746effd700 20 client.4432.objecter dump_active .. 0 homeless 2023-09-30T00:48:19.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.679+0000 7f747659c700 1 --2- 172.21.15.40:0/4107135346 >> v2:172.21.15.135:6800/116426 conn(0x7f7458068940 0x7f745806adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:19.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.679+0000 7f747659c700 10 monclient: get_auth_request con 0x7f7458068940 auth_method 0 2023-09-30T00:48:19.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.679+0000 7f747659c700 1 --2- 172.21.15.40:0/4107135346 >> v2:172.21.15.135:6800/116426 conn(0x7f7458068940 0x7f745806adf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f747813e9e0 tx=0x7f7460009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:19.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.679+0000 7f746effd700 10 client.4432.objecter ms_handle_connect 0x7f7458068940 2023-09-30T00:48:19.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.687+0000 7f746effd700 1 -- 172.21.15.40:0/4107135346 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f74680527f0 con 0x7f74780a5fb0 2023-09-30T00:48:19.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.687+0000 7f746effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:19.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.687+0000 7f746effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:19.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.940+0000 7f747d0a8700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:19.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.940+0000 7f747d0a8700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:19.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.940+0000 7f747d0a8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:19.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.940+0000 7f747d0a8700 1 -- 172.21.15.40:0/4107135346 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7478141520 con 0x7f74780a5fb0 2023-09-30T00:48:19.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.945+0000 7f746effd700 1 -- 172.21.15.40:0/4107135346 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7468055e10 con 0x7f74780a5fb0 2023-09-30T00:48:19.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.945+0000 7f746effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:19.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.945+0000 7f746effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:19.946 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:19.946 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:19.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.947+0000 7f747d0a8700 10 librados: watch_flush enter 2023-09-30T00:48:19.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.947+0000 7f747d0a8700 10 librados: watch_flush exit 2023-09-30T00:48:19.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.947+0000 7f747d0a8700 20 client.4432.objecter shutdown clearing up homeless session... 2023-09-30T00:48:19.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.947+0000 7f747d0a8700 10 client.4432.objecter successfully canceled tick 2023-09-30T00:48:19.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.947+0000 7f747d0a8700 1 -- 172.21.15.40:0/4107135346 >> v2:172.21.15.135:6800/116426 conn(0x7f7458068940 msgr2=0x7f745806adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:19.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.947+0000 7f747d0a8700 1 --2- 172.21.15.40:0/4107135346 >> v2:172.21.15.135:6800/116426 conn(0x7f7458068940 0x7f745806adf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f747813e9e0 tx=0x7f7460009040 comp rx=0 tx=0).stop 2023-09-30T00:48:19.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.948+0000 7f747d0a8700 10 monclient: shutdown 2023-09-30T00:48:19.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.948+0000 7f747d0a8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:19.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.948+0000 7f747d0a8700 1 -- 172.21.15.40:0/4107135346 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74780a5fb0 msgr2=0x7f747813e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:19.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.948+0000 7f747d0a8700 1 --2- 172.21.15.40:0/4107135346 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74780a5fb0 0x7f747813e130 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7468009ef0 tx=0x7f746800b040 comp rx=0 tx=0).stop 2023-09-30T00:48:19.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.948+0000 7f747d0a8700 1 -- 172.21.15.40:0/4107135346 shutdown_connections 2023-09-30T00:48:19.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.948+0000 7f747d0a8700 1 --2- 172.21.15.40:0/4107135346 >> v2:172.21.15.135:6800/116426 conn(0x7f7458068940 0x7f745806adf0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:19.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.948+0000 7f747d0a8700 1 --2- 172.21.15.40:0/4107135346 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f74780a5fb0 0x7f747813e130 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:19.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.948+0000 7f747d0a8700 1 --2- 172.21.15.40:0/4107135346 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f747813ac20 0x7f74780aa180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:19.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.948+0000 7f747d0a8700 1 --2- 172.21.15.40:0/4107135346 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f74780b1460 0x7f74780a9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:19.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.948+0000 7f747d0a8700 1 -- 172.21.15.40:0/4107135346 >> 172.21.15.40:0/4107135346 conn(0x7f74780631f0 msgr2=0x7f74780afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:19.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.948+0000 7f747d0a8700 1 -- 172.21.15.40:0/4107135346 shutdown_connections 2023-09-30T00:48:19.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.948+0000 7f747d0a8700 1 -- 172.21.15.40:0/4107135346 wait complete. 2023-09-30T00:48:19.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:19.948+0000 7f747d0a8700 1 librados: shutdown 2023-09-30T00:48:22.964 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:23.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.101+0000 7f122cda5700 10 monclient: get_monmap_and_config 2023-09-30T00:48:23.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.101+0000 7f122cda5700 10 monclient: build_initial_monmap 2023-09-30T00:48:23.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.101+0000 7f122cda5700 10 monclient: monmap: 2023-09-30T00:48:23.102 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:23.103 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:23.103 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:23.102341+0000 2023-09-30T00:48:23.103 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:23.102341+0000 2023-09-30T00:48:23.103 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:23.103 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:23.103 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:23.103 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:23.103 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:23.103 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:23.103 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:23.103 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:23.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.101+0000 7f122cda5700 1 Processor -- start 2023-09-30T00:48:23.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 1 -- start start 2023-09-30T00:48:23.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 10 monclient: init 2023-09-30T00:48:23.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:23.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-09-30T00:48:23.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f12280b1460 0x7f12280b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:23.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 10 monclient(hunting): picked mon.noname-c con 0x7f12280b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:23.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12280a5fb0 0x7f1228141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:23.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 10 monclient(hunting): picked mon.noname-b con 0x7f12280a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:23.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f12280af410 0x7f12280af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:23.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 10 monclient(hunting): picked mon.noname-a con 0x7f12280af410 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:23.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:23.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12280a71f0 con 0x7f12280af410 2023-09-30T00:48:23.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:23.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12280a74d0 con 0x7f12280b1460 2023-09-30T00:48:23.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:23.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12280a63c0 con 0x7f12280a5fb0 2023-09-30T00:48:23.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:23.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.102+0000 7f122cda5700 10 monclient(hunting): authenticate will time out at 5364.403923s 2023-09-30T00:48:23.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.106+0000 7f1226d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f12280af410 0x7f12280af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:23.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.138+0000 7f122659c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f12280b1460 0x7f12280b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:23.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.138+0000 7f122659c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f12280b1460 0x7f12280b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:34270/0 (socket says 172.21.15.40:34270) 2023-09-30T00:48:23.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.138+0000 7f122659c700 1 -- 172.21.15.40:0/628850053 learned_addr learned my addr 172.21.15.40:0/628850053 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.138+0000 7f122659c700 10 monclient(hunting): get_auth_request con 0x7f12280b1460 auth_method 0 2023-09-30T00:48:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.138+0000 7f122659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.138+0000 7f122659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.138+0000 7f122659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.139+0000 7f122659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.139+0000 7f122659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.139+0000 7f122659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.139+0000 7f1226d9d700 10 monclient(hunting): get_auth_request con 0x7f12280af410 auth_method 0 2023-09-30T00:48:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.139+0000 7f1226d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.139+0000 7f1226d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.139+0000 7f1226d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.141+0000 7f122659c700 10 monclient(hunting): handle_auth_done global_id 4435 payload 274 2023-09-30T00:48:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.141+0000 7f122659c700 1 -- 172.21.15.40:0/628850053 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12280a5fb0 msgr2=0x7f1228141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.141+0000 7f122659c700 1 --2- 172.21.15.40:0/628850053 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12280a5fb0 0x7f1228141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.141+0000 7f122659c700 1 -- 172.21.15.40:0/628850053 >> v2:172.21.15.40:3300/0 conn(0x7f12280af410 msgr2=0x7f12280af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:23.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.141+0000 7f122659c700 1 --2- 172.21.15.40:0/628850053 >> v2:172.21.15.40:3300/0 conn(0x7f12280af410 0x7f12280af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.141+0000 7f122659c700 10 monclient: _finish_hunting 0 2023-09-30T00:48:23.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.141+0000 7f122659c700 1 monclient: found mon.noname-c 2023-09-30T00:48:23.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.141+0000 7f122659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:23.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.141+0000 7f122659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:48:23.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.141+0000 7f122659c700 1 -- 172.21.15.40:0/628850053 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12280a56d0 con 0x7f12280b1460 2023-09-30T00:48:23.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.143+0000 7f122659c700 1 --2- 172.21.15.40:0/628850053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12280b1460 0x7f12280b1830 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f12280b1f90 tx=0x7f1218009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b9b7db08c479ba server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:23.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122559a700 1 -- 172.21.15.40:0/628850053 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121800b860 con 0x7f12280b1460 2023-09-30T00:48:23.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:23.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122559a700 10 monclient: dump: 2023-09-30T00:48:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:23.153 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122559a700 10 monclient: _finish_auth 0 2023-09-30T00:48:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122559a700 1 -- 172.21.15.40:0/628850053 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f121800b9c0 con 0x7f12280b1460 2023-09-30T00:48:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122cda5700 5 monclient: authenticate success, global_id 4435 2023-09-30T00:48:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122cda5700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122559a700 1 -- 172.21.15.40:0/628850053 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f121800bd60 con 0x7f12280b1460 2023-09-30T00:48:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122cda5700 4 set_mon_vals no callback set 2023-09-30T00:48:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.151+0000 7f122559a700 10 monclient: dump: 2023-09-30T00:48:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:23.155 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.152+0000 7f122cda5700 10 monclient: shutdown 2023-09-30T00:48:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.152+0000 7f122cda5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.152+0000 7f122cda5700 1 -- 172.21.15.40:0/628850053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12280b1460 msgr2=0x7f12280b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.152+0000 7f122cda5700 1 --2- 172.21.15.40:0/628850053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12280b1460 0x7f12280b1830 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f12280b1f90 tx=0x7f1218009250 comp rx=0 tx=0).stop 2023-09-30T00:48:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.152+0000 7f122cda5700 1 -- 172.21.15.40:0/628850053 shutdown_connections 2023-09-30T00:48:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.152+0000 7f122cda5700 1 --2- 172.21.15.40:0/628850053 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12280b1460 0x7f12280b1830 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.152+0000 7f122cda5700 1 --2- 172.21.15.40:0/628850053 >> v2:172.21.15.40:3300/0 conn(0x7f12280af410 0x7f12280af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.152+0000 7f122cda5700 1 --2- 172.21.15.40:0/628850053 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12280a5fb0 0x7f1228141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.152+0000 7f122cda5700 1 -- 172.21.15.40:0/628850053 >> 172.21.15.40:0/628850053 conn(0x7f12280631f0 msgr2=0x7f12280635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.152+0000 7f122cda5700 1 -- 172.21.15.40:0/628850053 shutdown_connections 2023-09-30T00:48:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.152+0000 7f122cda5700 1 -- 172.21.15.40:0/628850053 wait complete. 2023-09-30T00:48:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.153+0000 7f122cda5700 10 monclient: build_initial_monmap 2023-09-30T00:48:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.153+0000 7f122cda5700 10 monclient: monmap: 2023-09-30T00:48:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:23.161 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.154+0000 7f122cda5700 1 librados: starting msgr at 2023-09-30T00:48:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.154+0000 7f122cda5700 1 librados: starting objecter 2023-09-30T00:48:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.154+0000 7f122cda5700 1 Processor -- start 2023-09-30T00:48:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.154+0000 7f122cda5700 1 -- start start 2023-09-30T00:48:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.154+0000 7f122cda5700 1 librados: setting wanted keys 2023-09-30T00:48:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.154+0000 7f122cda5700 1 librados: calling monclient init 2023-09-30T00:48:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.154+0000 7f122cda5700 10 monclient: init 2023-09-30T00:48:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:48:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12280a5fb0 0x7f122813e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 10 monclient(hunting): picked mon.noname-c con 0x7f12280a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12280af410 0x7f12280a9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 10 monclient(hunting): picked mon.noname-a con 0x7f12280af410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12280b1460 0x7f122814d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 10 monclient(hunting): picked mon.noname-b con 0x7f12280b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12280a9df0 con 0x7f12280af410 2023-09-30T00:48:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12280a9a90 con 0x7f12280b1460 2023-09-30T00:48:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12280aa2f0 con 0x7f12280a5fb0 2023-09-30T00:48:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122cda5700 10 monclient(hunting): authenticate will time out at 5364.456309s 2023-09-30T00:48:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f1225d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12280af410 0x7f12280a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.155+0000 7f122659c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12280a5fb0 0x7f122813e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1226d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12280b1460 0x7f122814d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1225d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12280af410 0x7f12280a9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38278/0 (socket says 172.21.15.40:38278) 2023-09-30T00:48:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1225d9b700 1 -- 172.21.15.40:0/2333368838 learned_addr learned my addr 172.21.15.40:0/2333368838 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1225d9b700 10 monclient(hunting): get_auth_request con 0x7f12280af410 auth_method 0 2023-09-30T00:48:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1225d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1225d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1225d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f122659c700 10 monclient(hunting): get_auth_request con 0x7f12280a5fb0 auth_method 0 2023-09-30T00:48:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f122659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f122659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f122659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1226d9d700 10 monclient(hunting): get_auth_request con 0x7f12280b1460 auth_method 0 2023-09-30T00:48:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1226d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1226d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1226d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1225d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1225d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1225d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1226d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1226d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.156+0000 7f1226d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.157+0000 7f122659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.157+0000 7f122659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.157+0000 7f122659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.160+0000 7f1226d9d700 10 monclient(hunting): handle_auth_done global_id 4438 payload 824 2023-09-30T00:48:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.160+0000 7f1226d9d700 1 -- 172.21.15.40:0/2333368838 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12280a5fb0 msgr2=0x7f122813e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.160+0000 7f1226d9d700 1 --2- 172.21.15.40:0/2333368838 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12280a5fb0 0x7f122813e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.160+0000 7f1226d9d700 1 -- 172.21.15.40:0/2333368838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12280af410 msgr2=0x7f12280a9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.160+0000 7f1226d9d700 1 --2- 172.21.15.40:0/2333368838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12280af410 0x7f12280a9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.160+0000 7f1226d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:48:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.160+0000 7f1226d9d700 1 monclient: found mon.noname-b 2023-09-30T00:48:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.160+0000 7f1226d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.160+0000 7f1226d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.160+0000 7f1226d9d700 1 -- 172.21.15.40:0/2333368838 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12180094f0 con 0x7f12280b1460 2023-09-30T00:48:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.162+0000 7f1226d9d700 1 --2- 172.21.15.40:0/2333368838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12280b1460 0x7f122814d9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1220007ed0 tx=0x7f1220009850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.162+0000 7f12167fc700 10 client.?.objecter ms_handle_connect 0x7f12280b1460 2023-09-30T00:48:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.162+0000 7f12167fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.166+0000 7f12167fc700 1 -- 172.21.15.40:0/2333368838 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1220018070 con 0x7f12280b1460 2023-09-30T00:48:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.166+0000 7f12167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.166+0000 7f12167fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.166+0000 7f12167fc700 10 monclient: dump: 2023-09-30T00:48:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:23.169 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.166+0000 7f12167fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.166+0000 7f12167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.166+0000 7f12167fc700 1 -- 172.21.15.40:0/2333368838 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122000eb90 con 0x7f12280b1460 2023-09-30T00:48:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.166+0000 7f12167fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.166+0000 7f12167fc700 1 -- 172.21.15.40:0/2333368838 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1220014420 con 0x7f12280b1460 2023-09-30T00:48:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.166+0000 7f122cda5700 5 monclient: authenticate success, global_id 4438 2023-09-30T00:48:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f122cda5700 10 monclient: _renew_subs 2023-09-30T00:48:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f122cda5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f122cda5700 1 -- 172.21.15.40:0/2333368838 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f122813edf0 con 0x7f12280b1460 2023-09-30T00:48:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f12167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.166+0000 7f1224d99700 4 set_mon_vals no callback set 2023-09-30T00:48:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f12167fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f12167fc700 10 monclient: dump: 2023-09-30T00:48:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:23.171 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f122cda5700 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f122cda5700 10 monclient: _renew_subs 2023-09-30T00:48:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f122cda5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f122cda5700 1 -- 172.21.15.40:0/2333368838 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f122814e120 con 0x7f12280b1460 2023-09-30T00:48:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f122cda5700 1 librados: init done 2023-09-30T00:48:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f122cda5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f122cda5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f122cda5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.167+0000 7f122cda5700 1 -- 172.21.15.40:0/2333368838 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12280aa160 con 0x7f12280b1460 2023-09-30T00:48:23.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.414+0000 7f12167fc700 1 -- 172.21.15.40:0/2333368838 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1220003020 con 0x7f12280b1460 2023-09-30T00:48:23.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.415+0000 7f12167fc700 1 --2- 172.21.15.40:0/2333368838 >> v2:172.21.15.135:6800/116426 conn(0x7f1208068890 0x7f120806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:23.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.415+0000 7f122659c700 1 --2- 172.21.15.40:0/2333368838 >> v2:172.21.15.135:6800/116426 conn(0x7f1208068890 0x7f120806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:23.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.415+0000 7f122659c700 10 monclient: get_auth_request con 0x7f1208068890 auth_method 0 2023-09-30T00:48:23.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.416+0000 7f122659c700 1 --2- 172.21.15.40:0/2333368838 >> v2:172.21.15.135:6800/116426 conn(0x7f1208068890 0x7f120806ad40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f12280b1f90 tx=0x7f1218017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:23.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.416+0000 7f12167fc700 10 client.4438.objecter ms_handle_connect 0x7f1208068890 2023-09-30T00:48:23.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.419+0000 7f12167fc700 1 -- 172.21.15.40:0/2333368838 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1220056b40 con 0x7f12280b1460 2023-09-30T00:48:23.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.419+0000 7f12167fc700 10 client.4438.objecter ms_dispatch 0x7f1228001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:23.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.419+0000 7f12167fc700 3 client.4438.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:23.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.419+0000 7f12167fc700 3 client.4438.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:23.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.419+0000 7f12167fc700 20 client.4438.objecter dump_active .. 0 homeless 2023-09-30T00:48:23.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.440+0000 7f12167fc700 1 -- 172.21.15.40:0/2333368838 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1220052780 con 0x7f12280b1460 2023-09-30T00:48:23.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.440+0000 7f12167fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:23.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.440+0000 7f12167fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:23.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.723+0000 7f122cda5700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:23.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.723+0000 7f122cda5700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:23.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.723+0000 7f122cda5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:23.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.723+0000 7f122cda5700 1 -- 172.21.15.40:0/2333368838 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f12280a9f60 con 0x7f12280b1460 2023-09-30T00:48:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.730+0000 7f12167fc700 1 -- 172.21.15.40:0/2333368838 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1220055da0 con 0x7f12280b1460 2023-09-30T00:48:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.730+0000 7f12167fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:23.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.730+0000 7f12167fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:23.731 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:23.731 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:23.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 10 librados: watch_flush enter 2023-09-30T00:48:23.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 10 librados: watch_flush exit 2023-09-30T00:48:23.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 20 client.4438.objecter shutdown clearing up homeless session... 2023-09-30T00:48:23.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 10 client.4438.objecter successfully canceled tick 2023-09-30T00:48:23.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 1 -- 172.21.15.40:0/2333368838 >> v2:172.21.15.135:6800/116426 conn(0x7f1208068890 msgr2=0x7f120806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:23.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 1 --2- 172.21.15.40:0/2333368838 >> v2:172.21.15.135:6800/116426 conn(0x7f1208068890 0x7f120806ad40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f12280b1f90 tx=0x7f1218017000 comp rx=0 tx=0).stop 2023-09-30T00:48:23.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 10 monclient: shutdown 2023-09-30T00:48:23.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:23.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 1 -- 172.21.15.40:0/2333368838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12280b1460 msgr2=0x7f122814d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:23.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 1 --2- 172.21.15.40:0/2333368838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12280b1460 0x7f122814d9a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1220007ed0 tx=0x7f1220009850 comp rx=0 tx=0).stop 2023-09-30T00:48:23.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 1 -- 172.21.15.40:0/2333368838 shutdown_connections 2023-09-30T00:48:23.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 1 --2- 172.21.15.40:0/2333368838 >> v2:172.21.15.135:6800/116426 conn(0x7f1208068890 0x7f120806ad40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 1 --2- 172.21.15.40:0/2333368838 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f12280b1460 0x7f122814d9a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 1 --2- 172.21.15.40:0/2333368838 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f12280a5fb0 0x7f122813e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 1 --2- 172.21.15.40:0/2333368838 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12280af410 0x7f12280a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 1 -- 172.21.15.40:0/2333368838 >> 172.21.15.40:0/2333368838 conn(0x7f12280631f0 msgr2=0x7f12280635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:23.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 1 -- 172.21.15.40:0/2333368838 shutdown_connections 2023-09-30T00:48:23.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 1 -- 172.21.15.40:0/2333368838 wait complete. 2023-09-30T00:48:23.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:23.733+0000 7f122cda5700 1 librados: shutdown 2023-09-30T00:48:26.749 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.891+0000 7f56fba2e700 10 monclient: get_monmap_and_config 2023-09-30T00:48:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.891+0000 7f56fba2e700 10 monclient: build_initial_monmap 2023-09-30T00:48:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.891+0000 7f56fba2e700 10 monclient: monmap: 2023-09-30T00:48:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:26.892850+0000 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:26.892850+0000 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.892+0000 7f56fba2e700 1 Processor -- start 2023-09-30T00:48:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.892+0000 7f56fba2e700 1 -- start start 2023-09-30T00:48:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.892+0000 7f56fba2e700 10 monclient: init 2023-09-30T00:48:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-09-30T00:48:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f56f40b1460 0x7f56f40b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 10 monclient(hunting): picked mon.noname-c con 0x7f56f40b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56f40a5fb0 0x7f56f4141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 10 monclient(hunting): picked mon.noname-b con 0x7f56f40a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f56f40af410 0x7f56f40af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 10 monclient(hunting): picked mon.noname-a con 0x7f56f40af410 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f40a71f0 con 0x7f56f40af410 2023-09-30T00:48:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f40a74d0 con 0x7f56f40b1460 2023-09-30T00:48:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f40a63c0 con 0x7f56f40a5fb0 2023-09-30T00:48:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.893+0000 7f56fba2e700 10 monclient(hunting): authenticate will time out at 5368.194452s 2023-09-30T00:48:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f9fcb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f56f40af410 0x7f56f40af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f8fc9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56f40a5fb0 0x7f56f4141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f97ca700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f56f40b1460 0x7f56f40b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f9fcb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f56f40af410 0x7f56f40af890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38280/0 (socket says 172.21.15.40:38280) 2023-09-30T00:48:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f9fcb700 1 -- 172.21.15.40:0/2685789849 learned_addr learned my addr 172.21.15.40:0/2685789849 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f9fcb700 10 monclient(hunting): get_auth_request con 0x7f56f40af410 auth_method 0 2023-09-30T00:48:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f9fcb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f9fcb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f9fcb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f97ca700 10 monclient(hunting): get_auth_request con 0x7f56f40b1460 auth_method 0 2023-09-30T00:48:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f97ca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f97ca700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f97ca700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f8fc9700 10 monclient(hunting): get_auth_request con 0x7f56f40a5fb0 auth_method 0 2023-09-30T00:48:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f8fc9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f8fc9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.894+0000 7f56f8fc9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.895+0000 7f56f97ca700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.895+0000 7f56f97ca700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.895+0000 7f56f97ca700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.895+0000 7f56f9fcb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.895+0000 7f56f9fcb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.895+0000 7f56f9fcb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.895+0000 7f56f8fc9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.895+0000 7f56f8fc9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.895+0000 7f56f8fc9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.897+0000 7f56f97ca700 10 monclient(hunting): handle_auth_done global_id 4441 payload 274 2023-09-30T00:48:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.897+0000 7f56f97ca700 1 -- 172.21.15.40:0/2685789849 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56f40a5fb0 msgr2=0x7f56f4141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.897+0000 7f56f97ca700 1 --2- 172.21.15.40:0/2685789849 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56f40a5fb0 0x7f56f4141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.897+0000 7f56f97ca700 1 -- 172.21.15.40:0/2685789849 >> v2:172.21.15.40:3300/0 conn(0x7f56f40af410 msgr2=0x7f56f40af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.897+0000 7f56f97ca700 1 --2- 172.21.15.40:0/2685789849 >> v2:172.21.15.40:3300/0 conn(0x7f56f40af410 0x7f56f40af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.897+0000 7f56f97ca700 10 monclient: _finish_hunting 0 2023-09-30T00:48:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.897+0000 7f56f97ca700 1 monclient: found mon.noname-c 2023-09-30T00:48:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.897+0000 7f56f97ca700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.897+0000 7f56f97ca700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:48:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.897+0000 7f56f97ca700 1 -- 172.21.15.40:0/2685789849 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f40a56d0 con 0x7f56f40b1460 2023-09-30T00:48:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.899+0000 7f56f97ca700 1 --2- 172.21.15.40:0/2685789849 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56f40b1460 0x7f56f40b1830 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f56f40b1f90 tx=0x7f56ec009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9266e91afb0a9485 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.903+0000 7f56f3fff700 1 -- 172.21.15.40:0/2685789849 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec00e070 con 0x7f56f40b1460 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.903+0000 7f56f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.903+0000 7f56f3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.903+0000 7f56f3fff700 10 monclient: dump: 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.903+0000 7f56f3fff700 10 monclient: _finish_auth 0 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.903+0000 7f56f3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.903+0000 7f56f3fff700 1 -- 172.21.15.40:0/2685789849 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ec00bab0 con 0x7f56f40b1460 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.903+0000 7f56f3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56f3fff700 1 -- 172.21.15.40:0/2685789849 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec00be70 con 0x7f56f40b1460 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56f3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56f3fff700 10 monclient: dump: 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56fba2e700 5 monclient: authenticate success, global_id 4441 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56fba2e700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56fba2e700 4 set_mon_vals no callback set 2023-09-30T00:48:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56fba2e700 10 monclient: shutdown 2023-09-30T00:48:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56fba2e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56fba2e700 1 -- 172.21.15.40:0/2685789849 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56f40b1460 msgr2=0x7f56f40b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56fba2e700 1 --2- 172.21.15.40:0/2685789849 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56f40b1460 0x7f56f40b1830 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f56f40b1f90 tx=0x7f56ec009780 comp rx=0 tx=0).stop 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56fba2e700 1 -- 172.21.15.40:0/2685789849 shutdown_connections 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56fba2e700 1 --2- 172.21.15.40:0/2685789849 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56f40b1460 0x7f56f40b1830 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56fba2e700 1 --2- 172.21.15.40:0/2685789849 >> v2:172.21.15.40:3300/0 conn(0x7f56f40af410 0x7f56f40af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56fba2e700 1 --2- 172.21.15.40:0/2685789849 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56f40a5fb0 0x7f56f4141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56fba2e700 1 -- 172.21.15.40:0/2685789849 >> 172.21.15.40:0/2685789849 conn(0x7f56f40631f0 msgr2=0x7f56f40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56fba2e700 1 -- 172.21.15.40:0/2685789849 shutdown_connections 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.904+0000 7f56fba2e700 1 -- 172.21.15.40:0/2685789849 wait complete. 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.905+0000 7f56fba2e700 10 monclient: build_initial_monmap 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.905+0000 7f56fba2e700 10 monclient: monmap: 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.906+0000 7f56fba2e700 1 librados: starting msgr at 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.906+0000 7f56fba2e700 1 librados: starting objecter 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.906+0000 7f56fba2e700 1 Processor -- start 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.906+0000 7f56fba2e700 1 -- start start 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.906+0000 7f56fba2e700 1 librados: setting wanted keys 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.906+0000 7f56fba2e700 1 librados: calling monclient init 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.906+0000 7f56fba2e700 10 monclient: init 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56f40a5fb0 0x7f56f413e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 10 monclient(hunting): picked mon.noname-b con 0x7f56f40a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56f40af410 0x7f56f40a79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 10 monclient(hunting): picked mon.noname-c con 0x7f56f40af410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56f40b1460 0x7f56f40a7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 10 monclient(hunting): picked mon.noname-a con 0x7f56f40b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f40ab760 con 0x7f56f40b1460 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f40aba40 con 0x7f56f40a5fb0 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f40aa260 con 0x7f56f40af410 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.907+0000 7f56fba2e700 10 monclient(hunting): authenticate will time out at 5368.208582s 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f97ca700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56f40a5fb0 0x7f56f413e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f9fcb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56f40b1460 0x7f56f40a7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f8fc9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56f40af410 0x7f56f40a79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f97ca700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56f40a5fb0 0x7f56f413e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:34288/0 (socket says 172.21.15.40:34288) 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f97ca700 1 -- 172.21.15.40:0/1434868501 learned_addr learned my addr 172.21.15.40:0/1434868501 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f97ca700 10 monclient(hunting): get_auth_request con 0x7f56f40a5fb0 auth_method 0 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f97ca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f97ca700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f97ca700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f9fcb700 10 monclient(hunting): get_auth_request con 0x7f56f40b1460 auth_method 0 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f9fcb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f9fcb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f9fcb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f8fc9700 10 monclient(hunting): get_auth_request con 0x7f56f40af410 auth_method 0 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f8fc9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f8fc9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.908+0000 7f56f8fc9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.909+0000 7f56f97ca700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.909+0000 7f56f97ca700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.909+0000 7f56f97ca700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.909+0000 7f56f9fcb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.909+0000 7f56f9fcb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.909+0000 7f56f9fcb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.909+0000 7f56f8fc9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.909+0000 7f56f8fc9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.909+0000 7f56f8fc9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:26.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.912+0000 7f56f97ca700 10 monclient(hunting): handle_auth_done global_id 4444 payload 824 2023-09-30T00:48:26.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.912+0000 7f56f97ca700 1 -- 172.21.15.40:0/1434868501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56f40af410 msgr2=0x7f56f40a79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:26.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.912+0000 7f56f97ca700 1 --2- 172.21.15.40:0/1434868501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56f40af410 0x7f56f40a79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.912+0000 7f56f97ca700 1 -- 172.21.15.40:0/1434868501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56f40b1460 msgr2=0x7f56f40a7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.912+0000 7f56f97ca700 1 --2- 172.21.15.40:0/1434868501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56f40b1460 0x7f56f40a7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.912+0000 7f56f97ca700 10 monclient: _finish_hunting 0 2023-09-30T00:48:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.912+0000 7f56f97ca700 1 monclient: found mon.noname-b 2023-09-30T00:48:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.912+0000 7f56f97ca700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.912+0000 7f56f97ca700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.912+0000 7f56f97ca700 1 -- 172.21.15.40:0/1434868501 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec009a20 con 0x7f56f40a5fb0 2023-09-30T00:48:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.914+0000 7f56f97ca700 1 --2- 172.21.15.40:0/1434868501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56f40a5fb0 0x7f56f413e3b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009fd0 tx=0x7f56ec019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.914+0000 7f56f17fa700 10 client.?.objecter ms_handle_connect 0x7f56f40a5fb0 2023-09-30T00:48:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.914+0000 7f56f17fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.917+0000 7f56f17fa700 1 -- 172.21.15.40:0/1434868501 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec012a90 con 0x7f56f40a5fb0 2023-09-30T00:48:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.917+0000 7f56f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.917+0000 7f56f17fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.917+0000 7f56f17fa700 10 monclient: dump: 2023-09-30T00:48:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:26.921 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:26.921 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:26.921 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:26.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.917+0000 7f56f17fa700 10 monclient: _finish_auth 0 2023-09-30T00:48:26.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.917+0000 7f56f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:26.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.917+0000 7f56f17fa700 1 -- 172.21.15.40:0/1434868501 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ec012c10 con 0x7f56f40a5fb0 2023-09-30T00:48:26.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.917+0000 7f56fba2e700 5 monclient: authenticate success, global_id 4444 2023-09-30T00:48:26.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.917+0000 7f56fba2e700 10 monclient: _renew_subs 2023-09-30T00:48:26.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.917+0000 7f56fba2e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:26.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.917+0000 7f56fba2e700 1 -- 172.21.15.40:0/1434868501 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f40a9ef0 con 0x7f56f40a5fb0 2023-09-30T00:48:26.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.917+0000 7f56f17fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:26.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.917+0000 7f56f17fa700 1 -- 172.21.15.40:0/1434868501 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec027470 con 0x7f56f40a5fb0 2023-09-30T00:48:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.918+0000 7f56f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.918+0000 7f56f17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.918+0000 7f56f17fa700 10 monclient: dump: 2023-09-30T00:48:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:26.923 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.918+0000 7f56f37fe700 4 set_mon_vals no callback set 2023-09-30T00:48:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.918+0000 7f56fba2e700 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.918+0000 7f56fba2e700 10 monclient: _renew_subs 2023-09-30T00:48:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.918+0000 7f56fba2e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.918+0000 7f56fba2e700 1 -- 172.21.15.40:0/1434868501 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f40a98f0 con 0x7f56f40a5fb0 2023-09-30T00:48:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.918+0000 7f56fba2e700 1 librados: init done 2023-09-30T00:48:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.919+0000 7f56fba2e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.919+0000 7f56fba2e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.919+0000 7f56fba2e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:26.919+0000 7f56fba2e700 1 -- 172.21.15.40:0/1434868501 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f40ab8a0 con 0x7f56f40a5fb0 2023-09-30T00:48:27.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.145+0000 7f56f17fa700 1 -- 172.21.15.40:0/1434868501 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f56ec024070 con 0x7f56f40a5fb0 2023-09-30T00:48:27.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.145+0000 7f56f17fa700 1 --2- 172.21.15.40:0/1434868501 >> v2:172.21.15.135:6800/116426 conn(0x7f56e0068960 0x7f56e006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:27.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.146+0000 7f56f8fc9700 1 --2- 172.21.15.40:0/1434868501 >> v2:172.21.15.135:6800/116426 conn(0x7f56e0068960 0x7f56e006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:27.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.146+0000 7f56f8fc9700 10 monclient: get_auth_request con 0x7f56e0068960 auth_method 0 2023-09-30T00:48:27.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.146+0000 7f56f8fc9700 1 --2- 172.21.15.40:0/1434868501 >> v2:172.21.15.135:6800/116426 conn(0x7f56e0068960 0x7f56e006ae10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f56f40b3ff0 tx=0x7f56e80091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:27.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.146+0000 7f56f17fa700 10 client.4444.objecter ms_handle_connect 0x7f56e0068960 2023-09-30T00:48:27.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.169+0000 7f56f17fa700 1 -- 172.21.15.40:0/1434868501 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f56ec057f70 con 0x7f56f40a5fb0 2023-09-30T00:48:27.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.169+0000 7f56f17fa700 10 client.4444.objecter ms_dispatch 0x7f56f4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:27.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.169+0000 7f56f17fa700 3 client.4444.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:27.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.169+0000 7f56f17fa700 3 client.4444.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:27.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.169+0000 7f56f17fa700 20 client.4444.objecter dump_active .. 0 homeless 2023-09-30T00:48:27.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.179+0000 7f56f17fa700 1 -- 172.21.15.40:0/1434868501 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56ec053bb0 con 0x7f56f40a5fb0 2023-09-30T00:48:27.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.179+0000 7f56f17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:27.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.179+0000 7f56f17fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:27.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.430+0000 7f56fba2e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:27.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.430+0000 7f56fba2e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:27.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.430+0000 7f56fba2e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:27.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.430+0000 7f56fba2e700 1 -- 172.21.15.40:0/1434868501 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f56f40ab8a0 con 0x7f56f40a5fb0 2023-09-30T00:48:27.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.437+0000 7f56f17fa700 1 -- 172.21.15.40:0/1434868501 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f56f40ab8a0 con 0x7f56f40a5fb0 2023-09-30T00:48:27.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.437+0000 7f56f17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:27.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.437+0000 7f56f17fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:27.438 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:27.438 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:27.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 10 librados: watch_flush enter 2023-09-30T00:48:27.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 10 librados: watch_flush exit 2023-09-30T00:48:27.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 20 client.4444.objecter shutdown clearing up homeless session... 2023-09-30T00:48:27.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 10 client.4444.objecter successfully canceled tick 2023-09-30T00:48:27.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 1 -- 172.21.15.40:0/1434868501 >> v2:172.21.15.135:6800/116426 conn(0x7f56e0068960 msgr2=0x7f56e006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:27.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 1 --2- 172.21.15.40:0/1434868501 >> v2:172.21.15.135:6800/116426 conn(0x7f56e0068960 0x7f56e006ae10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f56f40b3ff0 tx=0x7f56e80091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:27.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 10 monclient: shutdown 2023-09-30T00:48:27.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:27.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 1 -- 172.21.15.40:0/1434868501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56f40a5fb0 msgr2=0x7f56f413e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:27.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 1 --2- 172.21.15.40:0/1434868501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56f40a5fb0 0x7f56f413e3b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009fd0 tx=0x7f56ec019000 comp rx=0 tx=0).stop 2023-09-30T00:48:27.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 1 -- 172.21.15.40:0/1434868501 shutdown_connections 2023-09-30T00:48:27.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 1 --2- 172.21.15.40:0/1434868501 >> v2:172.21.15.135:6800/116426 conn(0x7f56e0068960 0x7f56e006ae10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:27.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 1 --2- 172.21.15.40:0/1434868501 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56f40a5fb0 0x7f56f413e3b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:27.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 1 --2- 172.21.15.40:0/1434868501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56f40b1460 0x7f56f40a7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:27.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 1 --2- 172.21.15.40:0/1434868501 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56f40af410 0x7f56f40a79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:27.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 1 -- 172.21.15.40:0/1434868501 >> 172.21.15.40:0/1434868501 conn(0x7f56f40631f0 msgr2=0x7f56f40b28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:27.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 1 -- 172.21.15.40:0/1434868501 shutdown_connections 2023-09-30T00:48:27.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.440+0000 7f56fba2e700 1 -- 172.21.15.40:0/1434868501 wait complete. 2023-09-30T00:48:27.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:27.441+0000 7f56fba2e700 1 librados: shutdown 2023-09-30T00:48:27.774 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:02:03.485 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:27.827 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:03.513 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:27.828 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:03.515 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:27.832 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:02:03.543 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:27.851 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:03.538 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:27.868 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:02:03.579 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:27.903 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:02:03.615 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:27.913 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:02:03.625 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:27.938 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:02:03.650 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:30.455 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:30.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.598+0000 7f5bd408b700 10 monclient: get_monmap_and_config 2023-09-30T00:48:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.598+0000 7f5bd408b700 10 monclient: build_initial_monmap 2023-09-30T00:48:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.598+0000 7f5bd408b700 10 monclient: monmap: 2023-09-30T00:48:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:30.599426+0000 2023-09-30T00:48:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:30.599426+0000 2023-09-30T00:48:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:30.599 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:30.600 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:30.600 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.599+0000 7f5bd408b700 1 Processor -- start 2023-09-30T00:48:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.599+0000 7f5bd408b700 1 -- start start 2023-09-30T00:48:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.599+0000 7f5bd408b700 10 monclient: init 2023-09-30T00:48:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.599+0000 7f5bd408b700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.599+0000 7f5bd408b700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-09-30T00:48:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.599+0000 7f5bd408b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5bcc0b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:48:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.599+0000 7f5bd408b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5bcc0b1830 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.600+0000 7f5bd408b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5bcc0a5fb0 0x7f5bcc13b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.600+0000 7f5bd408b700 10 monclient(hunting): picked mon.noname-a con 0x7f5bcc0a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.600+0000 7f5bd408b700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bcc0a5b20 con 0x7f5bcc0b1460 2023-09-30T00:48:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.600+0000 7f5bd408b700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bcc0a71f0 con 0x7f5bcc0b1830 2023-09-30T00:48:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.600+0000 7f5bd408b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:30.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.600+0000 7f5bd408b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bcc141e40 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.600+0000 7f5bd408b700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:30.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.600+0000 7f5bd408b700 10 monclient(hunting): authenticate will time out at 5371.901073s 2023-09-30T00:48:30.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.600+0000 7f5bd1626700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f5bcc0b1830 0x7f5bcc13ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:32788/0 (socket says 172.21.15.40:32788) 2023-09-30T00:48:30.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.600+0000 7f5bd1626700 1 -- 172.21.15.40:0/931040471 learned_addr learned my addr 172.21.15.40:0/931040471 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:48:30.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.600+0000 7f5bd2628700 1 --2- 172.21.15.40:0/931040471 >> v2:172.21.15.40:3300/0 conn(0x7f5bcc0a5fb0 0x7f5bcc13b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:30.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.601+0000 7f5bd2628700 10 monclient(hunting): get_auth_request con 0x7f5bcc0a5fb0 auth_method 0 2023-09-30T00:48:30.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.601+0000 7f5bd2628700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:30.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.601+0000 7f5bd2628700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:30.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.601+0000 7f5bd2628700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.602+0000 7f5bd2628700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.602+0000 7f5bd2628700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.602+0000 7f5bd2628700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.604+0000 7f5bd0e25700 1 -- 172.21.15.40:0/931040471 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908257811 0 0) 0x7f5bcc0a71f0 con 0x7f5bcc0b1830 2023-09-30T00:48:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.604+0000 7f5bd0e25700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.604+0000 7f5bd0e25700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:30.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.604+0000 7f5bd0e25700 10 monclient(hunting): my global_id is 4447 2023-09-30T00:48:30.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.604+0000 7f5bd0e25700 1 -- 172.21.15.40:0/931040471 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb8003570 con 0x7f5bcc0b1830 2023-09-30T00:48:30.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.605+0000 7f5bd2628700 10 monclient(hunting): handle_auth_done global_id 4407 payload 274 2023-09-30T00:48:30.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.605+0000 7f5bd2628700 1 -- 172.21.15.40:0/931040471 >> v1:172.21.15.135:6789/0 conn(0x7f5bcc0b1830 legacy=0x7f5bcc13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:30.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.605+0000 7f5bd2628700 1 -- 172.21.15.40:0/931040471 >> v1:172.21.15.40:6789/0 conn(0x7f5bcc0b1460 legacy=0x7f5bcc141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:30.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.605+0000 7f5bd2628700 10 monclient: _finish_hunting 0 2023-09-30T00:48:30.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.605+0000 7f5bd2628700 1 monclient: found mon.noname-a 2023-09-30T00:48:30.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.605+0000 7f5bd2628700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:30.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.605+0000 7f5bd2628700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:30.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.605+0000 7f5bd2628700 1 -- 172.21.15.40:0/931040471 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc0a74d0 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.607+0000 7f5bd2628700 1 --2- 172.21.15.40:0/931040471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bcc0a5fb0 0x7f5bcc13b330 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4009e90 tx=0x7f5bc4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddaf91596c244441 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:30.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd0e25700 1 -- 172.21.15.40:0/931040471 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc400e070 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd0e25700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:30.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd0e25700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:30.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd0e25700 10 monclient: dump: 2023-09-30T00:48:30.615 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:30.615 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:30.615 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:30.615 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:30.615 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:30.615 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:30.615 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:30.616 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:30.616 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:30.616 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:30.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd0e25700 10 monclient: _finish_auth 0 2023-09-30T00:48:30.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd0e25700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:30.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd0e25700 1 -- 172.21.15.40:0/931040471 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc400bbb0 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd0e25700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:30.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd0e25700 1 -- 172.21.15.40:0/931040471 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4012430 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd0e25700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:30.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd0e25700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:30.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd0e25700 10 monclient: dump: 2023-09-30T00:48:30.616 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:30.617 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:30.617 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:30.617 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:30.617 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:30.617 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:30.617 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:30.617 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:30.617 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:30.617 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:30.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd408b700 5 monclient: authenticate success, global_id 4407 2023-09-30T00:48:30.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd408b700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:30.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.613+0000 7f5bd408b700 4 set_mon_vals no callback set 2023-09-30T00:48:30.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.614+0000 7f5bd408b700 10 monclient: shutdown 2023-09-30T00:48:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.614+0000 7f5bd408b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.614+0000 7f5bd408b700 1 -- 172.21.15.40:0/931040471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bcc0a5fb0 msgr2=0x7f5bcc13b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.614+0000 7f5bd408b700 1 --2- 172.21.15.40:0/931040471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bcc0a5fb0 0x7f5bcc13b330 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4009e90 tx=0x7f5bc4009780 comp rx=0 tx=0).stop 2023-09-30T00:48:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.614+0000 7f5bd408b700 1 -- 172.21.15.40:0/931040471 shutdown_connections 2023-09-30T00:48:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.614+0000 7f5bd408b700 1 --2- 172.21.15.40:0/931040471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bcc0a5fb0 0x7f5bcc13b330 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.614+0000 7f5bd408b700 1 -- 172.21.15.40:0/931040471 >> 172.21.15.40:0/931040471 conn(0x7f5bcc0631f0 msgr2=0x7f5bcc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.614+0000 7f5bd408b700 1 -- 172.21.15.40:0/931040471 shutdown_connections 2023-09-30T00:48:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.614+0000 7f5bd408b700 1 -- 172.21.15.40:0/931040471 wait complete. 2023-09-30T00:48:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.615+0000 7f5bd408b700 10 monclient: build_initial_monmap 2023-09-30T00:48:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.615+0000 7f5bd408b700 10 monclient: monmap: 2023-09-30T00:48:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:30.622 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.615+0000 7f5bd408b700 1 librados: starting msgr at 2023-09-30T00:48:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.615+0000 7f5bd408b700 1 librados: starting objecter 2023-09-30T00:48:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 1 Processor -- start 2023-09-30T00:48:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 1 -- start start 2023-09-30T00:48:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 1 librados: setting wanted keys 2023-09-30T00:48:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 1 librados: calling monclient init 2023-09-30T00:48:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 10 monclient: init 2023-09-30T00:48:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:48:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5bcc0a5fb0 0x7f5bcc13e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 10 monclient(hunting): picked mon.noname-b con 0x7f5bcc0a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bcc0b1460 0x7f5bcc0a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 10 monclient(hunting): picked mon.noname-a con 0x7f5bcc0b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bcc0b1830 0x7f5bcc0aa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 10 monclient(hunting): picked mon.noname-c con 0x7f5bcc0b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0a9ad0 con 0x7f5bcc0b1460 2023-09-30T00:48:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0ab7a0 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0aba80 con 0x7f5bcc0b1830 2023-09-30T00:48:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.616+0000 7f5bd408b700 10 monclient(hunting): authenticate will time out at 5371.917972s 2023-09-30T00:48:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.617+0000 7f5bd1626700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bcc0b1460 0x7f5bcc0a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.617+0000 7f5bd1e27700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5bcc0a5fb0 0x7f5bcc13e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.617+0000 7f5bd2628700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bcc0b1830 0x7f5bcc0aa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.617+0000 7f5bd1626700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bcc0b1460 0x7f5bcc0a9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38316/0 (socket says 172.21.15.40:38316) 2023-09-30T00:48:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.617+0000 7f5bd1626700 1 -- 172.21.15.40:0/1144476850 learned_addr learned my addr 172.21.15.40:0/1144476850 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.617+0000 7f5bd1626700 10 monclient(hunting): get_auth_request con 0x7f5bcc0b1460 auth_method 0 2023-09-30T00:48:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.617+0000 7f5bd1626700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.617+0000 7f5bd1626700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.617+0000 7f5bd1626700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.618+0000 7f5bd1e27700 10 monclient(hunting): get_auth_request con 0x7f5bcc0a5fb0 auth_method 0 2023-09-30T00:48:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.618+0000 7f5bd1e27700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.618+0000 7f5bd1e27700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.618+0000 7f5bd1e27700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.618+0000 7f5bd2628700 10 monclient(hunting): get_auth_request con 0x7f5bcc0b1830 auth_method 0 2023-09-30T00:48:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.618+0000 7f5bd2628700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.618+0000 7f5bd2628700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.618+0000 7f5bd2628700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.618+0000 7f5bd1626700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.618+0000 7f5bd1626700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.618+0000 7f5bd1626700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.619+0000 7f5bd1e27700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.619+0000 7f5bd1e27700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.619+0000 7f5bd1e27700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.619+0000 7f5bd2628700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.619+0000 7f5bd2628700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.619+0000 7f5bd2628700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.623+0000 7f5bd1e27700 10 monclient(hunting): handle_auth_done global_id 4450 payload 824 2023-09-30T00:48:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.623+0000 7f5bd1e27700 1 -- 172.21.15.40:0/1144476850 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bcc0b1830 msgr2=0x7f5bcc0aa420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.623+0000 7f5bd1e27700 1 --2- 172.21.15.40:0/1144476850 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bcc0b1830 0x7f5bcc0aa420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.623+0000 7f5bd1e27700 1 -- 172.21.15.40:0/1144476850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bcc0b1460 msgr2=0x7f5bcc0a9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.623+0000 7f5bd1e27700 1 --2- 172.21.15.40:0/1144476850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bcc0b1460 0x7f5bcc0a9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:30.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.623+0000 7f5bd1e27700 10 monclient: _finish_hunting 0 2023-09-30T00:48:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.623+0000 7f5bd1e27700 1 monclient: found mon.noname-b 2023-09-30T00:48:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.623+0000 7f5bd1e27700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.623+0000 7f5bd1e27700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.623+0000 7f5bd1e27700 1 -- 172.21.15.40:0/1144476850 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc4009a20 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.626+0000 7f5bd1e27700 1 --2- 172.21.15.40:0/1144476850 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5bcc0a5fb0 0x7f5bcc13e370 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5bc800b5e0 tx=0x7f5bc80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.626+0000 7f5bc1ffb700 10 client.?.objecter ms_handle_connect 0x7f5bcc0a5fb0 2023-09-30T00:48:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.626+0000 7f5bc1ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:30.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bc1ffb700 1 -- 172.21.15.40:0/1144476850 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc800f510 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bc1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:30.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bc1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:30.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bc1ffb700 10 monclient: dump: 2023-09-30T00:48:30.634 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:30.635 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bc1ffb700 10 monclient: _finish_auth 0 2023-09-30T00:48:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bc1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bc1ffb700 1 -- 172.21.15.40:0/1144476850 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc800f690 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bc1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bc1ffb700 1 -- 172.21.15.40:0/1144476850 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc800fa50 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bd408b700 5 monclient: authenticate success, global_id 4450 2023-09-30T00:48:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bc1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bc1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bc3fff700 4 set_mon_vals no callback set 2023-09-30T00:48:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bc1ffb700 10 monclient: dump: 2023-09-30T00:48:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:30.637 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bd408b700 10 monclient: _renew_subs 2023-09-30T00:48:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bd408b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bd408b700 1 -- 172.21.15.40:0/1144476850 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bcc13eff0 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bd408b700 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bd408b700 10 monclient: _renew_subs 2023-09-30T00:48:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bd408b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bd408b700 1 -- 172.21.15.40:0/1144476850 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bcc0a9420 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.632+0000 7f5bd408b700 1 librados: init done 2023-09-30T00:48:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.633+0000 7f5bd408b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.633+0000 7f5bd408b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.633+0000 7f5bd408b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.633+0000 7f5bd408b700 1 -- 172.21.15.40:0/1144476850 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bcc0a9c10 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.888+0000 7f5bc1ffb700 1 -- 172.21.15.40:0/1144476850 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5bc8024430 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.888+0000 7f5bc1ffb700 1 --2- 172.21.15.40:0/1144476850 >> v2:172.21.15.135:6800/116426 conn(0x7f5bb00689e0 0x7f5bb006ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:30.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.888+0000 7f5bc1ffb700 1 -- 172.21.15.40:0/1144476850 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5bc8084280 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.888+0000 7f5bc1ffb700 10 client.4450.objecter ms_dispatch 0x7f5bcc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:30.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.889+0000 7f5bc1ffb700 3 client.4450.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:30.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.889+0000 7f5bc1ffb700 3 client.4450.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:30.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.889+0000 7f5bd1626700 1 --2- 172.21.15.40:0/1144476850 >> v2:172.21.15.135:6800/116426 conn(0x7f5bb00689e0 0x7f5bb006ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:30.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.889+0000 7f5bd1626700 10 monclient: get_auth_request con 0x7f5bb00689e0 auth_method 0 2023-09-30T00:48:30.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.889+0000 7f5bc1ffb700 20 client.4450.objecter dump_active .. 0 homeless 2023-09-30T00:48:30.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.889+0000 7f5bc1ffb700 10 client.4450.objecter ms_handle_connect 0x7f5bb00689e0 2023-09-30T00:48:30.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.889+0000 7f5bd1626700 1 --2- 172.21.15.40:0/1144476850 >> v2:172.21.15.135:6800/116426 conn(0x7f5bb00689e0 0x7f5bb006ae90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc13e8d0 tx=0x7f5bbc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:30.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.897+0000 7f5bc1ffb700 1 -- 172.21.15.40:0/1144476850 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5bc8051ba0 con 0x7f5bcc0a5fb0 2023-09-30T00:48:30.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.897+0000 7f5bc1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:30.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:30.897+0000 7f5bc1ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.158+0000 7f5bd408b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.158+0000 7f5bd408b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.158+0000 7f5bd408b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:31.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.158+0000 7f5bd408b700 1 -- 172.21.15.40:0/1144476850 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5bcc064a00 con 0x7f5bcc0a5fb0 2023-09-30T00:48:31.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.166+0000 7f5bc1ffb700 1 -- 172.21.15.40:0/1144476850 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5bc80551c0 con 0x7f5bcc0a5fb0 2023-09-30T00:48:31.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.166+0000 7f5bc1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:31.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.166+0000 7f5bc1ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:31.167 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:31.167 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:31.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 10 librados: watch_flush enter 2023-09-30T00:48:31.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 10 librados: watch_flush exit 2023-09-30T00:48:31.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 20 client.4450.objecter shutdown clearing up homeless session... 2023-09-30T00:48:31.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 10 client.4450.objecter successfully canceled tick 2023-09-30T00:48:31.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 1 -- 172.21.15.40:0/1144476850 >> v2:172.21.15.135:6800/116426 conn(0x7f5bb00689e0 msgr2=0x7f5bb006ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 1 --2- 172.21.15.40:0/1144476850 >> v2:172.21.15.135:6800/116426 conn(0x7f5bb00689e0 0x7f5bb006ae90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc13e8d0 tx=0x7f5bbc009040 comp rx=0 tx=0).stop 2023-09-30T00:48:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 10 monclient: shutdown 2023-09-30T00:48:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 1 -- 172.21.15.40:0/1144476850 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5bcc0a5fb0 msgr2=0x7f5bcc13e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 1 --2- 172.21.15.40:0/1144476850 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5bcc0a5fb0 0x7f5bcc13e370 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5bc800b5e0 tx=0x7f5bc80090f0 comp rx=0 tx=0).stop 2023-09-30T00:48:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 1 -- 172.21.15.40:0/1144476850 shutdown_connections 2023-09-30T00:48:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 1 --2- 172.21.15.40:0/1144476850 >> v2:172.21.15.135:6800/116426 conn(0x7f5bb00689e0 0x7f5bb006ae90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 1 --2- 172.21.15.40:0/1144476850 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5bcc0a5fb0 0x7f5bcc13e370 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 1 --2- 172.21.15.40:0/1144476850 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5bcc0b1830 0x7f5bcc0aa420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 1 --2- 172.21.15.40:0/1144476850 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bcc0b1460 0x7f5bcc0a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 1 -- 172.21.15.40:0/1144476850 >> 172.21.15.40:0/1144476850 conn(0x7f5bcc0631f0 msgr2=0x7f5bcc0b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 1 -- 172.21.15.40:0/1144476850 shutdown_connections 2023-09-30T00:48:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 1 -- 172.21.15.40:0/1144476850 wait complete. 2023-09-30T00:48:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:31.169+0000 7f5bd408b700 1 librados: shutdown 2023-09-30T00:48:34.185 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.333+0000 7f5fe0dfd700 10 monclient: get_monmap_and_config 2023-09-30T00:48:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.333+0000 7f5fe0dfd700 10 monclient: build_initial_monmap 2023-09-30T00:48:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.333+0000 7f5fe0dfd700 10 monclient: monmap: 2023-09-30T00:48:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:34.334520+0000 2023-09-30T00:48:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:34.334520+0000 2023-09-30T00:48:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:34.335 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:34.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.334+0000 7f5fe0dfd700 1 Processor -- start 2023-09-30T00:48:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.334+0000 7f5fe0dfd700 1 -- start start 2023-09-30T00:48:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.334+0000 7f5fe0dfd700 10 monclient: init 2023-09-30T00:48:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fe0dfd700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fe0dfd700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-09-30T00:48:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fe0dfd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5fdc0a5710 0x7f5fdc0a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fe0dfd700 10 monclient(hunting): picked mon.noname-a con 0x7f5fdc0a5710 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fe0dfd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5fdc0a7b20 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fe0dfd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fdc13e8e0 0x7f5fdc13ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fe0dfd700 10 monclient(hunting): picked mon.noname-b con 0x7f5fdc13e8e0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fe0dfd700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fdc0a5ae0 con 0x7f5fdc0a7b20 2023-09-30T00:48:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fe0dfd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fe0dfd700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fdc0a7ff0 con 0x7f5fdc0a5710 2023-09-30T00:48:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fe0dfd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fe0dfd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc0aedc0 con 0x7f5fdc13e8e0 2023-09-30T00:48:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fe0dfd700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fe0dfd700 10 monclient(hunting): authenticate will time out at 5375.636329s 2023-09-30T00:48:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.335+0000 7f5fda59c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5fdc0a5710 0x7f5fdc0a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.336+0000 7f5fd9d9b700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f5fdc0a7b20 0x7f5fdc13e1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:37220/0 (socket says 172.21.15.40:37220) 2023-09-30T00:48:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.336+0000 7f5fd9d9b700 1 -- 172.21.15.40:0/507598271 learned_addr learned my addr 172.21.15.40:0/507598271 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:48:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.336+0000 7f5fda59c700 10 monclient(hunting): get_auth_request con 0x7f5fdc0a5710 auth_method 0 2023-09-30T00:48:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.336+0000 7f5fda59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.336+0000 7f5fda59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.336+0000 7f5fda59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:34.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.337+0000 7f5fda59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:34.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.337+0000 7f5fda59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:34.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.337+0000 7f5fda59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:34.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.339+0000 7f5fda59c700 10 monclient(hunting): handle_auth_done global_id 4413 payload 274 2023-09-30T00:48:34.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.339+0000 7f5fda59c700 1 -- 172.21.15.40:0/507598271 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fdc13e8e0 msgr2=0x7f5fdc13ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:34.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.339+0000 7f5fda59c700 1 --2- 172.21.15.40:0/507598271 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fdc13e8e0 0x7f5fdc13ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:34.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.339+0000 7f5fda59c700 1 -- 172.21.15.40:0/507598271 >> v1:172.21.15.135:6789/0 conn(0x7f5fdc0a7b20 legacy=0x7f5fdc13e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:34.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.339+0000 7f5fda59c700 10 monclient: _finish_hunting 0 2023-09-30T00:48:34.340 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.339+0000 7f5fda59c700 1 monclient: found mon.noname-a 2023-09-30T00:48:34.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.339+0000 7f5fda59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:34.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.339+0000 7f5fda59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:34.341 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.339+0000 7f5fda59c700 1 -- 172.21.15.40:0/507598271 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc0aef00 con 0x7f5fdc0a5710 2023-09-30T00:48:34.343 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.343+0000 7f5fda59c700 1 --2- 172.21.15.40:0/507598271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fdc0a5710 0x7f5fdc0a75e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc0098e0 tx=0x7f5fcc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d778e4e9278f3b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:34.352 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fd959a700 1 -- 172.21.15.40:0/507598271 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc009e80 con 0x7f5fdc0a5710 2023-09-30T00:48:34.352 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fd959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:34.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fd959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:34.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fd959a700 10 monclient: dump: 2023-09-30T00:48:34.353 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:34.353 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:34.353 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:34.353 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:34.353 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:34.353 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:34.353 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:34.353 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:34.353 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:34.353 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fd959a700 10 monclient: _finish_auth 0 2023-09-30T00:48:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fd959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fd959a700 1 -- 172.21.15.40:0/507598271 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fcc00b860 con 0x7f5fdc0a5710 2023-09-30T00:48:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fe0dfd700 5 monclient: authenticate success, global_id 4413 2023-09-30T00:48:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fe0dfd700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fd959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fe0dfd700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fe0dfd700 4 set_mon_vals no callback set 2023-09-30T00:48:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fd959a700 1 -- 172.21.15.40:0/507598271 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc00bbe0 con 0x7f5fdc0a5710 2023-09-30T00:48:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fd959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fd959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.351+0000 7f5fd959a700 10 monclient: dump: 2023-09-30T00:48:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:34.355 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:34.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.352+0000 7f5fe0dfd700 10 monclient: shutdown 2023-09-30T00:48:34.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.352+0000 7f5fe0dfd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:34.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.352+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/507598271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fdc0a5710 msgr2=0x7f5fdc0a75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:34.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.352+0000 7f5fe0dfd700 1 --2- 172.21.15.40:0/507598271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fdc0a5710 0x7f5fdc0a75e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc0098e0 tx=0x7f5fcc0091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:34.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.352+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/507598271 shutdown_connections 2023-09-30T00:48:34.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.352+0000 7f5fe0dfd700 1 --2- 172.21.15.40:0/507598271 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fdc13e8e0 0x7f5fdc13ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:34.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.352+0000 7f5fe0dfd700 1 --2- 172.21.15.40:0/507598271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fdc0a5710 0x7f5fdc0a75e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:34.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.352+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/507598271 >> 172.21.15.40:0/507598271 conn(0x7f5fdc069080 msgr2=0x7f5fdc0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.352+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/507598271 shutdown_connections 2023-09-30T00:48:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.352+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/507598271 wait complete. 2023-09-30T00:48:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.353+0000 7f5fe0dfd700 10 monclient: build_initial_monmap 2023-09-30T00:48:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.353+0000 7f5fe0dfd700 10 monclient: monmap: 2023-09-30T00:48:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:34.360 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.354+0000 7f5fe0dfd700 1 librados: starting msgr at 2023-09-30T00:48:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.354+0000 7f5fe0dfd700 1 librados: starting objecter 2023-09-30T00:48:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.354+0000 7f5fe0dfd700 1 Processor -- start 2023-09-30T00:48:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.354+0000 7f5fe0dfd700 1 -- start start 2023-09-30T00:48:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.354+0000 7f5fe0dfd700 1 librados: setting wanted keys 2023-09-30T00:48:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.354+0000 7f5fe0dfd700 1 librados: calling monclient init 2023-09-30T00:48:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.354+0000 7f5fe0dfd700 10 monclient: init 2023-09-30T00:48:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:48:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc0a5710 0x7f5fdc0a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 10 monclient(hunting): picked mon.noname-b con 0x7f5fdc0a5710 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fdc0a7b20 0x7f5fdc1514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 10 monclient(hunting): picked mon.noname-c con 0x7f5fdc0a7b20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fdc13e8e0 0x7f5fdc1519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 10 monclient(hunting): picked mon.noname-a con 0x7f5fdc13e8e0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc0aac70 con 0x7f5fdc13e8e0 2023-09-30T00:48:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc0ae180 con 0x7f5fdc0a5710 2023-09-30T00:48:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc0a8530 con 0x7f5fdc0a7b20 2023-09-30T00:48:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.355+0000 7f5fe0dfd700 10 monclient(hunting): authenticate will time out at 5375.656627s 2023-09-30T00:48:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.356+0000 7f5fd9d9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fdc0a7b20 0x7f5fdc1514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:34.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.356+0000 7f5fda59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc0a5710 0x7f5fdc0a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:34.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.356+0000 7f5fda59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc0a5710 0x7f5fdc0a90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:58554/0 (socket says 172.21.15.40:58554) 2023-09-30T00:48:34.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.356+0000 7f5fda59c700 1 -- 172.21.15.40:0/2848787778 learned_addr learned my addr 172.21.15.40:0/2848787778 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:34.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.356+0000 7f5fda59c700 10 monclient(hunting): get_auth_request con 0x7f5fdc0a5710 auth_method 0 2023-09-30T00:48:34.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.356+0000 7f5fda59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:34.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.356+0000 7f5fda59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:34.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.356+0000 7f5fda59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:34.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.357+0000 7f5fd9d9b700 10 monclient(hunting): get_auth_request con 0x7f5fdc0a7b20 auth_method 0 2023-09-30T00:48:34.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.357+0000 7f5fd9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:34.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.357+0000 7f5fd9d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:34.362 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.357+0000 7f5fd9d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:34.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.357+0000 7f5fda59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:34.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.357+0000 7f5fda59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:34.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.357+0000 7f5fda59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:34.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.357+0000 7f5fd9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:34.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.357+0000 7f5fd9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:34.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.357+0000 7f5fd9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.360+0000 7f5fda59c700 10 monclient(hunting): handle_auth_done global_id 4453 payload 824 2023-09-30T00:48:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.360+0000 7f5fda59c700 1 -- 172.21.15.40:0/2848787778 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fdc0a7b20 msgr2=0x7f5fdc1514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.360+0000 7f5fda59c700 1 --2- 172.21.15.40:0/2848787778 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fdc0a7b20 0x7f5fdc1514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.360+0000 7f5fda59c700 1 -- 172.21.15.40:0/2848787778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fdc13e8e0 msgr2=0x7f5fdc1519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.360+0000 7f5fda59c700 1 --2- 172.21.15.40:0/2848787778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fdc13e8e0 0x7f5fdc1519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.360+0000 7f5fda59c700 10 monclient: _finish_hunting 0 2023-09-30T00:48:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.360+0000 7f5fda59c700 1 monclient: found mon.noname-b 2023-09-30T00:48:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.360+0000 7f5fda59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.360+0000 7f5fda59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.360+0000 7f5fda59c700 1 -- 172.21.15.40:0/2848787778 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc009420 con 0x7f5fdc0a5710 2023-09-30T00:48:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.363+0000 7f5fda59c700 1 --2- 172.21.15.40:0/2848787778 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc0a5710 0x7f5fdc0a90d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc00b3c0 tx=0x7f5fcc017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.363+0000 7f5fca7fc700 10 client.?.objecter ms_handle_connect 0x7f5fdc0a5710 2023-09-30T00:48:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.363+0000 7f5fca7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:34.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fca7fc700 1 -- 172.21.15.40:0/2848787778 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc00bbe0 con 0x7f5fdc0a5710 2023-09-30T00:48:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fca7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fca7fc700 10 monclient: dump: 2023-09-30T00:48:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:34.368 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fca7fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fca7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fe0dfd700 5 monclient: authenticate success, global_id 4453 2023-09-30T00:48:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fe0dfd700 10 monclient: _renew_subs 2023-09-30T00:48:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fe0dfd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/2848787778 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc0a9610 con 0x7f5fdc0a5710 2023-09-30T00:48:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fe0dfd700 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fe0dfd700 10 monclient: _renew_subs 2023-09-30T00:48:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fe0dfd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/2848787778 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc1520e0 con 0x7f5fdc0a5710 2023-09-30T00:48:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.366+0000 7f5fe0dfd700 1 librados: init done 2023-09-30T00:48:34.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.367+0000 7f5fe0dfd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:34.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.367+0000 7f5fe0dfd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:34.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.367+0000 7f5fe0dfd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:34.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.367+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/2848787778 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc0a8750 con 0x7f5fdc0a5710 2023-09-30T00:48:34.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.656+0000 7f5fca7fc700 1 -- 172.21.15.40:0/2848787778 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fcc011630 con 0x7f5fdc0a5710 2023-09-30T00:48:34.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.656+0000 7f5fca7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.656+0000 7f5fca7fc700 1 -- 172.21.15.40:0/2848787778 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc006c50 con 0x7f5fdc0a5710 2023-09-30T00:48:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.656+0000 7f5fca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.656+0000 7f5fca7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.656+0000 7f5fca7fc700 10 monclient: dump: 2023-09-30T00:48:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:34.658 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:34.659 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:34.659 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:34.659 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:34.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.656+0000 7f5fd8d99700 4 set_mon_vals no callback set 2023-09-30T00:48:34.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.676+0000 7f5fca7fc700 1 -- 172.21.15.40:0/2848787778 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5fcc0117a0 con 0x7f5fdc0a5710 2023-09-30T00:48:34.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.677+0000 7f5fca7fc700 1 --2- 172.21.15.40:0/2848787778 >> v2:172.21.15.135:6800/116426 conn(0x7f5fbc068890 0x7f5fbc06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:34.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.677+0000 7f5fd9d9b700 1 --2- 172.21.15.40:0/2848787778 >> v2:172.21.15.135:6800/116426 conn(0x7f5fbc068890 0x7f5fbc06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:34.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.677+0000 7f5fd9d9b700 10 monclient: get_auth_request con 0x7f5fbc068890 auth_method 0 2023-09-30T00:48:34.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.678+0000 7f5fd9d9b700 1 --2- 172.21.15.40:0/2848787778 >> v2:172.21.15.135:6800/116426 conn(0x7f5fbc068890 0x7f5fbc06ad40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0aa7e0 tx=0x7f5fd0007eb0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:34.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.678+0000 7f5fca7fc700 10 client.4453.objecter ms_handle_connect 0x7f5fbc068890 2023-09-30T00:48:34.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.681+0000 7f5fca7fc700 1 -- 172.21.15.40:0/2848787778 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5fcc057b90 con 0x7f5fdc0a5710 2023-09-30T00:48:34.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.681+0000 7f5fca7fc700 10 client.4453.objecter ms_dispatch 0x7f5fdc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:34.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.681+0000 7f5fca7fc700 3 client.4453.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:34.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.681+0000 7f5fca7fc700 3 client.4453.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:34.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.681+0000 7f5fca7fc700 20 client.4453.objecter dump_active .. 0 homeless 2023-09-30T00:48:34.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.690+0000 7f5fca7fc700 1 -- 172.21.15.40:0/2848787778 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5fcc0537d0 con 0x7f5fdc0a5710 2023-09-30T00:48:34.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.690+0000 7f5fca7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:34.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.690+0000 7f5fca7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:34.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.947+0000 7f5fe0dfd700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:34.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.947+0000 7f5fe0dfd700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:34.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.947+0000 7f5fe0dfd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:34.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.947+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/2848787778 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5fdc072b90 con 0x7f5fdc0a5710 2023-09-30T00:48:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.952+0000 7f5fca7fc700 1 -- 172.21.15.40:0/2848787778 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5fcc056df0 con 0x7f5fdc0a5710 2023-09-30T00:48:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.952+0000 7f5fca7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:34.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.952+0000 7f5fca7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:34.952 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:34.952 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 10 librados: watch_flush enter 2023-09-30T00:48:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 10 librados: watch_flush exit 2023-09-30T00:48:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 20 client.4453.objecter shutdown clearing up homeless session... 2023-09-30T00:48:34.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 10 client.4453.objecter successfully canceled tick 2023-09-30T00:48:34.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/2848787778 >> v2:172.21.15.135:6800/116426 conn(0x7f5fbc068890 msgr2=0x7f5fbc06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:34.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 1 --2- 172.21.15.40:0/2848787778 >> v2:172.21.15.135:6800/116426 conn(0x7f5fbc068890 0x7f5fbc06ad40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0aa7e0 tx=0x7f5fd0007eb0 comp rx=0 tx=0).stop 2023-09-30T00:48:34.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 10 monclient: shutdown 2023-09-30T00:48:34.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:34.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/2848787778 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc0a5710 msgr2=0x7f5fdc0a90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:34.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 1 --2- 172.21.15.40:0/2848787778 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc0a5710 0x7f5fdc0a90d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc00b3c0 tx=0x7f5fcc017000 comp rx=0 tx=0).stop 2023-09-30T00:48:34.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/2848787778 shutdown_connections 2023-09-30T00:48:34.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 1 --2- 172.21.15.40:0/2848787778 >> v2:172.21.15.135:6800/116426 conn(0x7f5fbc068890 0x7f5fbc06ad40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:34.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 1 --2- 172.21.15.40:0/2848787778 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5fdc0a5710 0x7f5fdc0a90d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:34.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 1 --2- 172.21.15.40:0/2848787778 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fdc13e8e0 0x7f5fdc1519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:34.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 1 --2- 172.21.15.40:0/2848787778 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5fdc0a7b20 0x7f5fdc1514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:34.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/2848787778 >> 172.21.15.40:0/2848787778 conn(0x7f5fdc069080 msgr2=0x7f5fdc143130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:34.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/2848787778 shutdown_connections 2023-09-30T00:48:34.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 1 -- 172.21.15.40:0/2848787778 wait complete. 2023-09-30T00:48:34.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:34.954+0000 7f5fe0dfd700 1 librados: shutdown 2023-09-30T00:48:37.968 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:38.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.099+0000 7f459dc57700 10 monclient: get_monmap_and_config 2023-09-30T00:48:38.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.099+0000 7f459dc57700 10 monclient: build_initial_monmap 2023-09-30T00:48:38.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.099+0000 7f459dc57700 10 monclient: monmap: 2023-09-30T00:48:38.100 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:38.100560+0000 2023-09-30T00:48:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:38.100560+0000 2023-09-30T00:48:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:38.101 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.100+0000 7f459dc57700 1 Processor -- start 2023-09-30T00:48:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.100+0000 7f459dc57700 1 -- start start 2023-09-30T00:48:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.100+0000 7f459dc57700 10 monclient: init 2023-09-30T00:48:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.100+0000 7f459dc57700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-09-30T00:48:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f45980b1460 0x7f45980b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 10 monclient(hunting): picked mon.noname-c con 0x7f45980b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:38.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f45980a5fb0 0x7f4598141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 10 monclient(hunting): picked mon.noname-a con 0x7f45980a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45980af410 0x7f45980af870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 10 monclient(hunting): picked mon.noname-b con 0x7f45980af410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45980a71f0 con 0x7f45980a5fb0 2023-09-30T00:48:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45980a74d0 con 0x7f45980b1460 2023-09-30T00:48:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45980a63c0 con 0x7f45980af410 2023-09-30T00:48:38.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f459dc57700 10 monclient(hunting): authenticate will time out at 5379.402097s 2023-09-30T00:48:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f4597fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45980af410 0x7f45980af870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f4596ffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f45980a5fb0 0x7f4598141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.101+0000 7f45977fe700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f45980b1460 0x7f45980b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f45977fe700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f45980b1460 0x7f45980b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:58558/0 (socket says 172.21.15.40:58558) 2023-09-30T00:48:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f45977fe700 1 -- 172.21.15.40:0/1954050863 learned_addr learned my addr 172.21.15.40:0/1954050863 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f45977fe700 10 monclient(hunting): get_auth_request con 0x7f45980b1460 auth_method 0 2023-09-30T00:48:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f45977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f45977fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:38.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f45977fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f4597fff700 10 monclient(hunting): get_auth_request con 0x7f45980af410 auth_method 0 2023-09-30T00:48:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f4597fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f4597fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f4597fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f4596ffd700 10 monclient(hunting): get_auth_request con 0x7f45980a5fb0 auth_method 0 2023-09-30T00:48:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f4596ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f4596ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f4596ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f45977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f45977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f45977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f4597fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f4597fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.102+0000 7f4597fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.103+0000 7f4596ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.103+0000 7f4596ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.103+0000 7f4596ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.105+0000 7f45977fe700 10 monclient(hunting): handle_auth_done global_id 4456 payload 274 2023-09-30T00:48:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.105+0000 7f45977fe700 1 -- 172.21.15.40:0/1954050863 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45980af410 msgr2=0x7f45980af870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.105+0000 7f45977fe700 1 --2- 172.21.15.40:0/1954050863 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45980af410 0x7f45980af870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.105+0000 7f45977fe700 1 -- 172.21.15.40:0/1954050863 >> v2:172.21.15.40:3300/0 conn(0x7f45980a5fb0 msgr2=0x7f4598141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.105+0000 7f45977fe700 1 --2- 172.21.15.40:0/1954050863 >> v2:172.21.15.40:3300/0 conn(0x7f45980a5fb0 0x7f4598141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.105+0000 7f45977fe700 10 monclient: _finish_hunting 0 2023-09-30T00:48:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.105+0000 7f45977fe700 1 monclient: found mon.noname-c 2023-09-30T00:48:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.105+0000 7f45977fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.105+0000 7f45977fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:48:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.105+0000 7f45977fe700 1 -- 172.21.15.40:0/1954050863 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45980a56d0 con 0x7f45980b1460 2023-09-30T00:48:38.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.106+0000 7f45977fe700 1 --2- 172.21.15.40:0/1954050863 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45980b1460 0x7f45980b1830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f45980b1f90 tx=0x7f4588009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=231a88c8b9ae204 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f45967fc700 1 -- 172.21.15.40:0/1954050863 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458800b640 con 0x7f45980b1460 2023-09-30T00:48:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f45967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f45967fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f45967fc700 10 monclient: dump: 2023-09-30T00:48:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:38.112 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f45967fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f45967fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f45967fc700 1 -- 172.21.15.40:0/1954050863 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458800b7a0 con 0x7f45980b1460 2023-09-30T00:48:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f45967fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f45967fc700 1 -- 172.21.15.40:0/1954050863 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458800bb60 con 0x7f45980b1460 2023-09-30T00:48:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f459dc57700 5 monclient: authenticate success, global_id 4456 2023-09-30T00:48:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f459dc57700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f459dc57700 4 set_mon_vals no callback set 2023-09-30T00:48:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f45967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f45967fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f45967fc700 10 monclient: dump: 2023-09-30T00:48:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:38.114 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:38.114 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:38.114 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:38.114 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:38.114 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:38.114 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:38.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f459dc57700 10 monclient: shutdown 2023-09-30T00:48:38.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f459dc57700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:38.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f459dc57700 1 -- 172.21.15.40:0/1954050863 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45980b1460 msgr2=0x7f45980b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:38.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f459dc57700 1 --2- 172.21.15.40:0/1954050863 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45980b1460 0x7f45980b1830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f45980b1f90 tx=0x7f4588009250 comp rx=0 tx=0).stop 2023-09-30T00:48:38.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f459dc57700 1 -- 172.21.15.40:0/1954050863 shutdown_connections 2023-09-30T00:48:38.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f459dc57700 1 --2- 172.21.15.40:0/1954050863 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45980b1460 0x7f45980b1830 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:38.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f459dc57700 1 --2- 172.21.15.40:0/1954050863 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45980af410 0x7f45980af870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:38.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f459dc57700 1 --2- 172.21.15.40:0/1954050863 >> v2:172.21.15.40:3300/0 conn(0x7f45980a5fb0 0x7f4598141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:38.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f459dc57700 1 -- 172.21.15.40:0/1954050863 >> 172.21.15.40:0/1954050863 conn(0x7f45980631f0 msgr2=0x7f45980635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:38.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.111+0000 7f459dc57700 1 -- 172.21.15.40:0/1954050863 shutdown_connections 2023-09-30T00:48:38.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.112+0000 7f459dc57700 1 -- 172.21.15.40:0/1954050863 wait complete. 2023-09-30T00:48:38.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.112+0000 7f459dc57700 10 monclient: build_initial_monmap 2023-09-30T00:48:38.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.112+0000 7f459dc57700 10 monclient: monmap: 2023-09-30T00:48:38.118 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:38.118 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:38.118 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:38.118 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:38.118 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:38.118 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:38.118 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:38.118 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:38.118 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:38.118 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:38.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.112+0000 7f459dc57700 1 librados: starting msgr at 2023-09-30T00:48:38.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.112+0000 7f459dc57700 1 librados: starting objecter 2023-09-30T00:48:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 1 Processor -- start 2023-09-30T00:48:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 1 -- start start 2023-09-30T00:48:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 1 librados: setting wanted keys 2023-09-30T00:48:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 1 librados: calling monclient init 2023-09-30T00:48:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 10 monclient: init 2023-09-30T00:48:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:48:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45980a5fb0 0x7f459813e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 10 monclient(hunting): picked mon.noname-c con 0x7f45980a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45980af410 0x7f45980a7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:38.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 10 monclient(hunting): picked mon.noname-b con 0x7f45980af410 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45980b1460 0x7f45980a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 10 monclient(hunting): picked mon.noname-a con 0x7f45980b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45980a7ab0 con 0x7f45980b1460 2023-09-30T00:48:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45980a9010 con 0x7f45980af410 2023-09-30T00:48:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45980a91d0 con 0x7f45980a5fb0 2023-09-30T00:48:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.113+0000 7f459dc57700 10 monclient(hunting): authenticate will time out at 5379.414946s 2023-09-30T00:48:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.114+0000 7f4597fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45980b1460 0x7f45980a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.114+0000 7f45977fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45980a5fb0 0x7f459813e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.114+0000 7f4596ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45980af410 0x7f45980a7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.114+0000 7f4597fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45980b1460 0x7f45980a8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55702/0 (socket says 172.21.15.40:55702) 2023-09-30T00:48:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.114+0000 7f4597fff700 1 -- 172.21.15.40:0/738999000 learned_addr learned my addr 172.21.15.40:0/738999000 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.114+0000 7f4597fff700 10 monclient(hunting): get_auth_request con 0x7f45980b1460 auth_method 0 2023-09-30T00:48:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.114+0000 7f4597fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.114+0000 7f4597fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.114+0000 7f4597fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.114+0000 7f45977fe700 10 monclient(hunting): get_auth_request con 0x7f45980a5fb0 auth_method 0 2023-09-30T00:48:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.114+0000 7f45977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.114+0000 7f45977fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:38.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.114+0000 7f45977fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:38.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.115+0000 7f4596ffd700 10 monclient(hunting): get_auth_request con 0x7f45980af410 auth_method 0 2023-09-30T00:48:38.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.115+0000 7f4596ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:38.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.115+0000 7f4596ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:38.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.115+0000 7f4596ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:38.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.115+0000 7f4597fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:38.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.115+0000 7f4597fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:38.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.115+0000 7f4597fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:38.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.115+0000 7f4596ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:38.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.115+0000 7f4596ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:38.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.115+0000 7f4596ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.115+0000 7f45977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.115+0000 7f45977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.115+0000 7f45977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.118+0000 7f4596ffd700 10 monclient(hunting): handle_auth_done global_id 4459 payload 824 2023-09-30T00:48:38.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.118+0000 7f4596ffd700 1 -- 172.21.15.40:0/738999000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45980a5fb0 msgr2=0x7f459813e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:38.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.118+0000 7f4596ffd700 1 --2- 172.21.15.40:0/738999000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45980a5fb0 0x7f459813e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:38.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.118+0000 7f4596ffd700 1 -- 172.21.15.40:0/738999000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45980b1460 msgr2=0x7f45980a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:38.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.118+0000 7f4596ffd700 1 --2- 172.21.15.40:0/738999000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45980b1460 0x7f45980a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:38.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.118+0000 7f4596ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:48:38.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.118+0000 7f4596ffd700 1 monclient: found mon.noname-b 2023-09-30T00:48:38.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.118+0000 7f4596ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:38.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.118+0000 7f4596ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:38.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.118+0000 7f4596ffd700 1 -- 172.21.15.40:0/738999000 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45880094f0 con 0x7f45980af410 2023-09-30T00:48:38.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.120+0000 7f4596ffd700 1 --2- 172.21.15.40:0/738999000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45980af410 0x7f45980a7f40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f458000ea50 tx=0x7f45800091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:38.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.120+0000 7f458f7fe700 10 client.?.objecter ms_handle_connect 0x7f45980af410 2023-09-30T00:48:38.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.120+0000 7f458f7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f458f7fe700 1 -- 172.21.15.40:0/738999000 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458000ee90 con 0x7f45980af410 2023-09-30T00:48:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f458f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f458f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f458f7fe700 10 monclient: dump: 2023-09-30T00:48:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:38.126 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:38.127 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:38.127 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:38.127 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:38.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f458f7fe700 10 monclient: _finish_auth 0 2023-09-30T00:48:38.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f458f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:38.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f458f7fe700 1 -- 172.21.15.40:0/738999000 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458000c7f0 con 0x7f45980af410 2023-09-30T00:48:38.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f459dc57700 5 monclient: authenticate success, global_id 4459 2023-09-30T00:48:38.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f458f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:38.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f458f7fe700 1 -- 172.21.15.40:0/738999000 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458000cb90 con 0x7f45980af410 2023-09-30T00:48:38.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f458f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:38.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f458f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f458f7fe700 10 monclient: dump: 2023-09-30T00:48:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:38.128 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f459dc57700 10 monclient: _renew_subs 2023-09-30T00:48:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f459dc57700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f459dc57700 1 -- 172.21.15.40:0/738999000 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459814dd20 con 0x7f45980af410 2023-09-30T00:48:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f4595ffb700 4 set_mon_vals no callback set 2023-09-30T00:48:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f459dc57700 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f459dc57700 10 monclient: _renew_subs 2023-09-30T00:48:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f459dc57700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f459dc57700 1 -- 172.21.15.40:0/738999000 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459814e240 con 0x7f45980af410 2023-09-30T00:48:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.124+0000 7f459dc57700 1 librados: init done 2023-09-30T00:48:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.125+0000 7f459dc57700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.125+0000 7f459dc57700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.125+0000 7f459dc57700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.125+0000 7f459dc57700 1 -- 172.21.15.40:0/738999000 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4598064a00 con 0x7f45980af410 2023-09-30T00:48:38.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.150+0000 7f458f7fe700 1 -- 172.21.15.40:0/738999000 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f4580013470 con 0x7f45980af410 2023-09-30T00:48:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.150+0000 7f458f7fe700 1 --2- 172.21.15.40:0/738999000 >> v2:172.21.15.135:6800/116426 conn(0x7f4578068a10 0x7f457806aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.151+0000 7f458f7fe700 1 -- 172.21.15.40:0/738999000 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f4580084790 con 0x7f45980af410 2023-09-30T00:48:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.151+0000 7f458f7fe700 10 client.4459.objecter ms_dispatch 0x7f4598001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.151+0000 7f458f7fe700 3 client.4459.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.151+0000 7f458f7fe700 3 client.4459.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.151+0000 7f458f7fe700 20 client.4459.objecter dump_active .. 0 homeless 2023-09-30T00:48:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.151+0000 7f45977fe700 1 --2- 172.21.15.40:0/738999000 >> v2:172.21.15.135:6800/116426 conn(0x7f4578068a10 0x7f457806aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.151+0000 7f45977fe700 10 monclient: get_auth_request con 0x7f4578068a10 auth_method 0 2023-09-30T00:48:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.151+0000 7f458f7fe700 10 client.4459.objecter ms_handle_connect 0x7f4578068a10 2023-09-30T00:48:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.151+0000 7f45977fe700 1 --2- 172.21.15.40:0/738999000 >> v2:172.21.15.135:6800/116426 conn(0x7f4578068a10 0x7f457806aec0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f45980b1f90 tx=0x7f4588017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:38.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.390+0000 7f458f7fe700 1 -- 172.21.15.40:0/738999000 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f45800520b0 con 0x7f45980af410 2023-09-30T00:48:38.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.390+0000 7f458f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:38.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.390+0000 7f458f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:38.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.656+0000 7f459dc57700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:38.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.656+0000 7f459dc57700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:38.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.656+0000 7f459dc57700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:38.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.656+0000 7f459dc57700 1 -- 172.21.15.40:0/738999000 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f45980b3f60 con 0x7f45980af410 2023-09-30T00:48:38.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.663+0000 7f458f7fe700 1 -- 172.21.15.40:0/738999000 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f45800556d0 con 0x7f45980af410 2023-09-30T00:48:38.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.663+0000 7f458f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:38.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.663+0000 7f458f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:38.663 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:38.664 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:38.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 10 librados: watch_flush enter 2023-09-30T00:48:38.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 10 librados: watch_flush exit 2023-09-30T00:48:38.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 20 client.4459.objecter shutdown clearing up homeless session... 2023-09-30T00:48:38.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 10 client.4459.objecter successfully canceled tick 2023-09-30T00:48:38.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 1 -- 172.21.15.40:0/738999000 >> v2:172.21.15.135:6800/116426 conn(0x7f4578068a10 msgr2=0x7f457806aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:38.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 1 --2- 172.21.15.40:0/738999000 >> v2:172.21.15.135:6800/116426 conn(0x7f4578068a10 0x7f457806aec0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f45980b1f90 tx=0x7f4588017000 comp rx=0 tx=0).stop 2023-09-30T00:48:38.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 10 monclient: shutdown 2023-09-30T00:48:38.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:38.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 1 -- 172.21.15.40:0/738999000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45980af410 msgr2=0x7f45980a7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:38.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 1 --2- 172.21.15.40:0/738999000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45980af410 0x7f45980a7f40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f458000ea50 tx=0x7f45800091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:38.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 1 -- 172.21.15.40:0/738999000 shutdown_connections 2023-09-30T00:48:38.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 1 --2- 172.21.15.40:0/738999000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45980b1460 0x7f45980a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:38.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 1 --2- 172.21.15.40:0/738999000 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f45980a5fb0 0x7f459813e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:38.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 1 --2- 172.21.15.40:0/738999000 >> v2:172.21.15.135:6800/116426 conn(0x7f4578068a10 0x7f457806aec0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:38.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 1 --2- 172.21.15.40:0/738999000 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f45980af410 0x7f45980a7f40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:38.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 1 -- 172.21.15.40:0/738999000 >> 172.21.15.40:0/738999000 conn(0x7f45980631f0 msgr2=0x7f459813b200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:38.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 1 -- 172.21.15.40:0/738999000 shutdown_connections 2023-09-30T00:48:38.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 1 -- 172.21.15.40:0/738999000 wait complete. 2023-09-30T00:48:38.668 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:38.665+0000 7f459dc57700 1 librados: shutdown 2023-09-30T00:48:41.680 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:41.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.818+0000 7ffb09f2f700 10 monclient: get_monmap_and_config 2023-09-30T00:48:41.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.818+0000 7ffb09f2f700 10 monclient: build_initial_monmap 2023-09-30T00:48:41.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.818+0000 7ffb09f2f700 10 monclient: monmap: 2023-09-30T00:48:41.819 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:41.819 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:41.819 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:41.819260+0000 2023-09-30T00:48:41.819 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:41.819260+0000 2023-09-30T00:48:41.819 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:41.819 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:41.819 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:41.819 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:41.820 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:41.820 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:41.820 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:41.820 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 1 Processor -- start 2023-09-30T00:48:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 1 -- start start 2023-09-30T00:48:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 10 monclient: init 2023-09-30T00:48:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2023-09-30T00:48:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffb040b1460 0x7ffb040b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 10 monclient(hunting): picked mon.noname-a con 0x7ffb040b1460 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:41.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb040a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ffb0413ac20 0x7ffb0413b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 10 monclient(hunting): picked mon.noname-c con 0x7ffb0413ac20 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb040a5b20 con 0x7ffb040a5fb0 2023-09-30T00:48:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb040a71f0 con 0x7ffb040b1460 2023-09-30T00:48:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb040a74d0 con 0x7ffb0413ac20 2023-09-30T00:48:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.819+0000 7ffb09f2f700 10 monclient(hunting): authenticate will time out at 5383.120841s 2023-09-30T00:48:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.820+0000 7ffb037fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffb040b1460 0x7ffb040b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:41.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.820+0000 7ffb03fff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7ffb0413ac20 0x7ffb0413b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.820+0000 7ffb037fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffb040b1460 0x7ffb040b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44084/0 (socket says 172.21.15.40:44084) 2023-09-30T00:48:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.820+0000 7ffb037fe700 1 -- 172.21.15.40:0/4030347484 learned_addr learned my addr 172.21.15.40:0/4030347484 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.820+0000 7ffb037fe700 10 monclient(hunting): get_auth_request con 0x7ffb040b1460 auth_method 0 2023-09-30T00:48:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.820+0000 7ffb037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.820+0000 7ffb037fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.820+0000 7ffb037fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.821+0000 7ffb03fff700 10 monclient(hunting): get_auth_request con 0x7ffb0413ac20 auth_method 0 2023-09-30T00:48:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.821+0000 7ffb03fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.821+0000 7ffb03fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.821+0000 7ffb03fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.821+0000 7ffb037fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:41.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.821+0000 7ffb037fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.821+0000 7ffb037fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.821+0000 7ffb03fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.821+0000 7ffb03fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:41.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.821+0000 7ffb03fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.824+0000 7ffb037fe700 10 monclient(hunting): handle_auth_done global_id 4422 payload 274 2023-09-30T00:48:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.824+0000 7ffb037fe700 1 -- 172.21.15.40:0/4030347484 >> v2:172.21.15.135:3300/0 conn(0x7ffb0413ac20 msgr2=0x7ffb0413b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.824+0000 7ffb037fe700 1 --2- 172.21.15.40:0/4030347484 >> v2:172.21.15.135:3300/0 conn(0x7ffb0413ac20 0x7ffb0413b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.824+0000 7ffb037fe700 1 -- 172.21.15.40:0/4030347484 >> v1:172.21.15.135:6789/0 conn(0x7ffb040a5fb0 legacy=0x7ffb04141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.824+0000 7ffb037fe700 10 monclient: _finish_hunting 0 2023-09-30T00:48:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.824+0000 7ffb037fe700 1 monclient: found mon.noname-a 2023-09-30T00:48:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.824+0000 7ffb037fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.824+0000 7ffb037fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:41.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.824+0000 7ffb037fe700 1 -- 172.21.15.40:0/4030347484 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb040a63c0 con 0x7ffb040b1460 2023-09-30T00:48:41.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.826+0000 7ffb037fe700 1 --2- 172.21.15.40:0/4030347484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb040b1460 0x7ffb040b1830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffaf40099b0 tx=0x7ffaf4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68d4c3052d84416 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:41.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb027fc700 1 -- 172.21.15.40:0/4030347484 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4009ea0 con 0x7ffb040b1460 2023-09-30T00:48:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb027fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb027fc700 10 monclient: dump: 2023-09-30T00:48:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:41.832 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:41.833 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:41.833 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:41.833 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:41.833 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:41.833 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:41.833 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:41.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb027fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:41.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:41.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb027fc700 1 -- 172.21.15.40:0/4030347484 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf400b7e0 con 0x7ffb040b1460 2023-09-30T00:48:41.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb027fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:41.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb027fc700 1 -- 172.21.15.40:0/4030347484 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf400bb80 con 0x7ffb040b1460 2023-09-30T00:48:41.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb09f2f700 5 monclient: authenticate success, global_id 4422 2023-09-30T00:48:41.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb09f2f700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:41.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb09f2f700 4 set_mon_vals no callback set 2023-09-30T00:48:41.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:41.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb027fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:41.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb027fc700 10 monclient: dump: 2023-09-30T00:48:41.835 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:41.835 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:41.835 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:41.835 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:41.835 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:41.836 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:41.836 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:41.836 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:41.836 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:41.836 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:41.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb09f2f700 10 monclient: shutdown 2023-09-30T00:48:41.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb09f2f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:41.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb09f2f700 1 -- 172.21.15.40:0/4030347484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb040b1460 msgr2=0x7ffb040b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:41.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.831+0000 7ffb09f2f700 1 --2- 172.21.15.40:0/4030347484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb040b1460 0x7ffb040b1830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffaf40099b0 tx=0x7ffaf4009250 comp rx=0 tx=0).stop 2023-09-30T00:48:41.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.832+0000 7ffb09f2f700 1 -- 172.21.15.40:0/4030347484 shutdown_connections 2023-09-30T00:48:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.832+0000 7ffb09f2f700 1 --2- 172.21.15.40:0/4030347484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb040b1460 0x7ffb040b1830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.832+0000 7ffb09f2f700 1 --2- 172.21.15.40:0/4030347484 >> v2:172.21.15.135:3300/0 conn(0x7ffb0413ac20 0x7ffb0413b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.832+0000 7ffb09f2f700 1 -- 172.21.15.40:0/4030347484 >> 172.21.15.40:0/4030347484 conn(0x7ffb040631f0 msgr2=0x7ffb040635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.832+0000 7ffb09f2f700 1 -- 172.21.15.40:0/4030347484 shutdown_connections 2023-09-30T00:48:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.832+0000 7ffb09f2f700 1 -- 172.21.15.40:0/4030347484 wait complete. 2023-09-30T00:48:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.832+0000 7ffb09f2f700 10 monclient: build_initial_monmap 2023-09-30T00:48:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.832+0000 7ffb09f2f700 10 monclient: monmap: 2023-09-30T00:48:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:41.842 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:41.842 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:41.842 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:41.842 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:41.842 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:41.842 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:41.842 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:41.842 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:41.843 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:41.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.833+0000 7ffb09f2f700 1 librados: starting msgr at 2023-09-30T00:48:41.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.833+0000 7ffb09f2f700 1 librados: starting objecter 2023-09-30T00:48:41.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.833+0000 7ffb09f2f700 1 Processor -- start 2023-09-30T00:48:41.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.833+0000 7ffb09f2f700 1 -- start start 2023-09-30T00:48:41.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.833+0000 7ffb09f2f700 1 librados: setting wanted keys 2023-09-30T00:48:41.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.833+0000 7ffb09f2f700 1 librados: calling monclient init 2023-09-30T00:48:41.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.833+0000 7ffb09f2f700 10 monclient: init 2023-09-30T00:48:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:48:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb040a5fb0 0x7ffb0413e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 10 monclient(hunting): picked mon.noname-b con 0x7ffb040a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb040b1460 0x7ffb040a7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 10 monclient(hunting): picked mon.noname-a con 0x7ffb040b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffb0413ac20 0x7ffb040a7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 10 monclient(hunting): picked mon.noname-c con 0x7ffb0413ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb040ab7a0 con 0x7ffb040b1460 2023-09-30T00:48:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb040aba80 con 0x7ffb040a5fb0 2023-09-30T00:48:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb040aa2a0 con 0x7ffb0413ac20 2023-09-30T00:48:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb09f2f700 10 monclient(hunting): authenticate will time out at 5383.135413s 2023-09-30T00:48:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.834+0000 7ffb02ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb040b1460 0x7ffb040a7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb03fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffb0413ac20 0x7ffb040a7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb037fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb040a5fb0 0x7ffb0413e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb02ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb040b1460 0x7ffb040a7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44092/0 (socket says 172.21.15.40:44092) 2023-09-30T00:48:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb02ffd700 1 -- 172.21.15.40:0/565787172 learned_addr learned my addr 172.21.15.40:0/565787172 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb02ffd700 10 monclient(hunting): get_auth_request con 0x7ffb040b1460 auth_method 0 2023-09-30T00:48:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb02ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb02ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb02ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb037fe700 10 monclient(hunting): get_auth_request con 0x7ffb040a5fb0 auth_method 0 2023-09-30T00:48:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb037fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb037fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb037fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb03fff700 10 monclient(hunting): get_auth_request con 0x7ffb0413ac20 auth_method 0 2023-09-30T00:48:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb03fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb03fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb03fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb037fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb037fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb037fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb02ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.835+0000 7ffb02ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.836+0000 7ffb02ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.836+0000 7ffb03fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.836+0000 7ffb03fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.836+0000 7ffb03fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.839+0000 7ffb037fe700 10 monclient(hunting): handle_auth_done global_id 4465 payload 824 2023-09-30T00:48:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.839+0000 7ffb037fe700 1 -- 172.21.15.40:0/565787172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffb0413ac20 msgr2=0x7ffb040a7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.839+0000 7ffb037fe700 1 --2- 172.21.15.40:0/565787172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffb0413ac20 0x7ffb040a7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.839+0000 7ffb037fe700 1 -- 172.21.15.40:0/565787172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb040b1460 msgr2=0x7ffb040a7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:41.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.839+0000 7ffb037fe700 1 --2- 172.21.15.40:0/565787172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb040b1460 0x7ffb040a7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:41.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.839+0000 7ffb037fe700 10 monclient: _finish_hunting 0 2023-09-30T00:48:41.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.839+0000 7ffb037fe700 1 monclient: found mon.noname-b 2023-09-30T00:48:41.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.839+0000 7ffb037fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:41.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.839+0000 7ffb037fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:41.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.839+0000 7ffb037fe700 1 -- 172.21.15.40:0/565787172 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf40094f0 con 0x7ffb040a5fb0 2023-09-30T00:48:41.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.841+0000 7ffb037fe700 1 --2- 172.21.15.40:0/565787172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb040a5fb0 0x7ffb0413e420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ffb040a6b60 tx=0x7ffaf4016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:41.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.841+0000 7ffafb7fe700 10 client.?.objecter ms_handle_connect 0x7ffb040a5fb0 2023-09-30T00:48:41.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.841+0000 7ffafb7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:41.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffafb7fe700 1 -- 172.21.15.40:0/565787172 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4011b70 con 0x7ffb040a5fb0 2023-09-30T00:48:41.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffafb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:41.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffafb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:41.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffafb7fe700 10 monclient: dump: 2023-09-30T00:48:41.853 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:41.853 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:41.853 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:41.853 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:41.853 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:41.854 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffafb7fe700 10 monclient: _finish_auth 0 2023-09-30T00:48:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffafb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffafb7fe700 1 -- 172.21.15.40:0/565787172 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf4020070 con 0x7ffb040a5fb0 2023-09-30T00:48:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffb09f2f700 5 monclient: authenticate success, global_id 4465 2023-09-30T00:48:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffafb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffb09f2f700 10 monclient: _renew_subs 2023-09-30T00:48:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffafb7fe700 1 -- 172.21.15.40:0/565787172 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf401b450 con 0x7ffb040a5fb0 2023-09-30T00:48:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffb09f2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffb09f2f700 1 -- 172.21.15.40:0/565787172 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb040aa7c0 con 0x7ffb040a5fb0 2023-09-30T00:48:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffafb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffb01ffb700 4 set_mon_vals no callback set 2023-09-30T00:48:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffafb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffafb7fe700 10 monclient: dump: 2023-09-30T00:48:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:41.857 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:41.857 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:41.857 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:41.857 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:41.857 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:41.857 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:41.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffb09f2f700 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:41.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffb09f2f700 10 monclient: _renew_subs 2023-09-30T00:48:41.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffb09f2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:41.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffb09f2f700 1 -- 172.21.15.40:0/565787172 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb040a9930 con 0x7ffb040a5fb0 2023-09-30T00:48:41.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.845+0000 7ffb09f2f700 1 librados: init done 2023-09-30T00:48:41.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.846+0000 7ffb09f2f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:41.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.846+0000 7ffb09f2f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:41.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.846+0000 7ffb09f2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:41.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:41.846+0000 7ffb09f2f700 1 -- 172.21.15.40:0/565787172 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0406f610 con 0x7ffb040a5fb0 2023-09-30T00:48:42.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.100+0000 7ffafb7fe700 1 -- 172.21.15.40:0/565787172 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ffaf4022030 con 0x7ffb040a5fb0 2023-09-30T00:48:42.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.100+0000 7ffafb7fe700 1 --2- 172.21.15.40:0/565787172 >> v2:172.21.15.135:6800/116426 conn(0x7ffae40689e0 0x7ffae406ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:42.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.100+0000 7ffafb7fe700 1 -- 172.21.15.40:0/565787172 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ffaf4086030 con 0x7ffb040a5fb0 2023-09-30T00:48:42.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.100+0000 7ffafb7fe700 10 client.4465.objecter ms_dispatch 0x7ffb04001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:42.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.100+0000 7ffafb7fe700 3 client.4465.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:42.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.100+0000 7ffafb7fe700 3 client.4465.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:42.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.100+0000 7ffafb7fe700 20 client.4465.objecter dump_active .. 0 homeless 2023-09-30T00:48:42.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.100+0000 7ffb02ffd700 1 --2- 172.21.15.40:0/565787172 >> v2:172.21.15.135:6800/116426 conn(0x7ffae40689e0 0x7ffae406ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:42.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.100+0000 7ffb02ffd700 10 monclient: get_auth_request con 0x7ffae40689e0 auth_method 0 2023-09-30T00:48:42.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.101+0000 7ffb02ffd700 1 --2- 172.21.15.40:0/565787172 >> v2:172.21.15.135:6800/116426 conn(0x7ffae40689e0 0x7ffae406ae90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffb0413eca0 tx=0x7ffaec007f60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:42.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.101+0000 7ffafb7fe700 10 client.4465.objecter ms_handle_connect 0x7ffae40689e0 2023-09-30T00:48:42.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.108+0000 7ffafb7fe700 1 -- 172.21.15.40:0/565787172 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ffaf4053950 con 0x7ffb040a5fb0 2023-09-30T00:48:42.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.109+0000 7ffafb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:42.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.109+0000 7ffafb7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:42.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.367+0000 7ffb09f2f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:42.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.367+0000 7ffb09f2f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:42.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.367+0000 7ffb09f2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:42.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.367+0000 7ffb09f2f700 1 -- 172.21.15.40:0/565787172 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ffb040abbc0 con 0x7ffb040a5fb0 2023-09-30T00:48:42.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.372+0000 7ffafb7fe700 1 -- 172.21.15.40:0/565787172 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ffb040abbc0 con 0x7ffb040a5fb0 2023-09-30T00:48:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.372+0000 7ffafb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:42.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.372+0000 7ffafb7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:42.373 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:42.373 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.374+0000 7ffb09f2f700 10 librados: watch_flush enter 2023-09-30T00:48:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.374+0000 7ffb09f2f700 10 librados: watch_flush exit 2023-09-30T00:48:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.374+0000 7ffb09f2f700 20 client.4465.objecter shutdown clearing up homeless session... 2023-09-30T00:48:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.374+0000 7ffb09f2f700 10 client.4465.objecter successfully canceled tick 2023-09-30T00:48:42.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.374+0000 7ffb09f2f700 1 -- 172.21.15.40:0/565787172 >> v2:172.21.15.135:6800/116426 conn(0x7ffae40689e0 msgr2=0x7ffae406ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.374+0000 7ffb09f2f700 1 --2- 172.21.15.40:0/565787172 >> v2:172.21.15.135:6800/116426 conn(0x7ffae40689e0 0x7ffae406ae90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffb0413eca0 tx=0x7ffaec007f60 comp rx=0 tx=0).stop 2023-09-30T00:48:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.374+0000 7ffb09f2f700 10 monclient: shutdown 2023-09-30T00:48:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.374+0000 7ffb09f2f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.374+0000 7ffb09f2f700 1 -- 172.21.15.40:0/565787172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb040a5fb0 msgr2=0x7ffb0413e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.374+0000 7ffb09f2f700 1 --2- 172.21.15.40:0/565787172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb040a5fb0 0x7ffb0413e420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ffb040a6b60 tx=0x7ffaf4016000 comp rx=0 tx=0).stop 2023-09-30T00:48:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.375+0000 7ffb09f2f700 1 -- 172.21.15.40:0/565787172 shutdown_connections 2023-09-30T00:48:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.375+0000 7ffb09f2f700 1 --2- 172.21.15.40:0/565787172 >> v2:172.21.15.135:6800/116426 conn(0x7ffae40689e0 0x7ffae406ae90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.375+0000 7ffb09f2f700 1 --2- 172.21.15.40:0/565787172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ffb040a5fb0 0x7ffb0413e420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.375+0000 7ffb09f2f700 1 --2- 172.21.15.40:0/565787172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ffb0413ac20 0x7ffb040a7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:42.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.375+0000 7ffb09f2f700 1 --2- 172.21.15.40:0/565787172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb040b1460 0x7ffb040a7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.375+0000 7ffb09f2f700 1 -- 172.21.15.40:0/565787172 >> 172.21.15.40:0/565787172 conn(0x7ffb040631f0 msgr2=0x7ffb040b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.375+0000 7ffb09f2f700 1 -- 172.21.15.40:0/565787172 shutdown_connections 2023-09-30T00:48:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.375+0000 7ffb09f2f700 1 -- 172.21.15.40:0/565787172 wait complete. 2023-09-30T00:48:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:42.375+0000 7ffb09f2f700 1 librados: shutdown 2023-09-30T00:48:45.390 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.530+0000 7fa751129700 10 monclient: get_monmap_and_config 2023-09-30T00:48:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.530+0000 7fa751129700 10 monclient: build_initial_monmap 2023-09-30T00:48:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.530+0000 7fa751129700 10 monclient: monmap: 2023-09-30T00:48:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:45.531 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:45.531265+0000 2023-09-30T00:48:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:45.531265+0000 2023-09-30T00:48:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:45.532 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:45.533 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:45.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.530+0000 7fa751129700 1 Processor -- start 2023-09-30T00:48:45.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.530+0000 7fa751129700 1 -- start start 2023-09-30T00:48:45.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.530+0000 7fa751129700 10 monclient: init 2023-09-30T00:48:45.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.531+0000 7fa751129700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.531+0000 7fa751129700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-09-30T00:48:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.531+0000 7fa751129700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa74c0b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:48:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.531+0000 7fa751129700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa74c0b1830 0x7fa74c0a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.531+0000 7fa751129700 10 monclient(hunting): picked mon.noname-c con 0x7fa74c0b1830 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.531+0000 7fa751129700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa74c13ac20 0x7fa74c13b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.531+0000 7fa751129700 10 monclient(hunting): picked mon.noname-b con 0x7fa74c13ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.531+0000 7fa751129700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74c0a5b20 con 0x7fa74c0b1460 2023-09-30T00:48:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.531+0000 7fa751129700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.531+0000 7fa751129700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74c141e40 con 0x7fa74c0b1830 2023-09-30T00:48:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.531+0000 7fa751129700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:45.535 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.531+0000 7fa751129700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c0a64f0 con 0x7fa74c13ac20 2023-09-30T00:48:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.531+0000 7fa751129700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.531+0000 7fa751129700 10 monclient(hunting): authenticate will time out at 5386.832601s 2023-09-30T00:48:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.532+0000 7fa74ad9d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa74c0b1460 0x7fa74c141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57866/0 (socket says 172.21.15.40:57866) 2023-09-30T00:48:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.532+0000 7fa74ad9d700 1 -- 172.21.15.40:0/3487307182 learned_addr learned my addr 172.21.15.40:0/3487307182 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:48:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.532+0000 7fa74b59e700 1 --2- 172.21.15.40:0/3487307182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa74c13ac20 0x7fa74c13b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.532+0000 7fa74b59e700 10 monclient(hunting): get_auth_request con 0x7fa74c13ac20 auth_method 0 2023-09-30T00:48:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.532+0000 7fa74b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.532+0000 7fa74b59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.532+0000 7fa74b59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.533+0000 7fa74b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.533+0000 7fa74b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:45.536 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.533+0000 7fa74b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.536+0000 7fa74b59e700 10 monclient(hunting): handle_auth_done global_id 4373 payload 274 2023-09-30T00:48:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.536+0000 7fa74b59e700 1 -- 172.21.15.40:0/3487307182 >> v2:172.21.15.135:3300/0 conn(0x7fa74c0b1830 msgr2=0x7fa74c0a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.536+0000 7fa74b59e700 1 --2- 172.21.15.40:0/3487307182 >> v2:172.21.15.135:3300/0 conn(0x7fa74c0b1830 0x7fa74c0a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:45.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.536+0000 7fa74b59e700 1 -- 172.21.15.40:0/3487307182 >> v1:172.21.15.40:6789/0 conn(0x7fa74c0b1460 legacy=0x7fa74c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.536+0000 7fa74b59e700 10 monclient: _finish_hunting 0 2023-09-30T00:48:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.536+0000 7fa74b59e700 1 monclient: found mon.noname-b 2023-09-30T00:48:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.536+0000 7fa74b59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.536+0000 7fa74b59e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:48:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.536+0000 7fa74b59e700 1 -- 172.21.15.40:0/3487307182 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c0a71f0 con 0x7fa74c13ac20 2023-09-30T00:48:45.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.538+0000 7fa74b59e700 1 --2- 172.21.15.40:0/3487307182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa74c13ac20 0x7fa74c13b080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa744009e60 tx=0x7fa744009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee4c6594fa1462a4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa749d9b700 1 -- 172.21.15.40:0/3487307182 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74400e070 con 0x7fa74c13ac20 2023-09-30T00:48:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa749d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa749d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa749d9b700 10 monclient: dump: 2023-09-30T00:48:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:45.545 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:45.546 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:45.546 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:45.546 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:45.546 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa749d9b700 10 monclient: _finish_auth 0 2023-09-30T00:48:45.546 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa749d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:45.546 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa749d9b700 1 -- 172.21.15.40:0/3487307182 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74400bc30 con 0x7fa74c13ac20 2023-09-30T00:48:45.546 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa749d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:45.546 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa749d9b700 1 -- 172.21.15.40:0/3487307182 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa744012430 con 0x7fa74c13ac20 2023-09-30T00:48:45.546 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa749d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:45.547 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa749d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:45.547 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa749d9b700 10 monclient: dump: 2023-09-30T00:48:45.547 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:45.547 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:45.547 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:45.547 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:45.547 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:45.547 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:45.547 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:45.547 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:45.547 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:45.548 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa751129700 5 monclient: authenticate success, global_id 4373 2023-09-30T00:48:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa751129700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.543+0000 7fa751129700 4 set_mon_vals no callback set 2023-09-30T00:48:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.544+0000 7fa751129700 10 monclient: shutdown 2023-09-30T00:48:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.544+0000 7fa751129700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.544+0000 7fa751129700 1 -- 172.21.15.40:0/3487307182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa74c13ac20 msgr2=0x7fa74c13b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.544+0000 7fa751129700 1 --2- 172.21.15.40:0/3487307182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa74c13ac20 0x7fa74c13b080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa744009e60 tx=0x7fa744009780 comp rx=0 tx=0).stop 2023-09-30T00:48:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.544+0000 7fa751129700 1 -- 172.21.15.40:0/3487307182 shutdown_connections 2023-09-30T00:48:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.544+0000 7fa751129700 1 --2- 172.21.15.40:0/3487307182 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa74c13ac20 0x7fa74c13b080 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.544+0000 7fa751129700 1 --2- 172.21.15.40:0/3487307182 >> v2:172.21.15.135:3300/0 conn(0x7fa74c0b1830 0x7fa74c0a5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:45.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.544+0000 7fa751129700 1 -- 172.21.15.40:0/3487307182 >> 172.21.15.40:0/3487307182 conn(0x7fa74c0631f0 msgr2=0x7fa74c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:45.549 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.544+0000 7fa751129700 1 -- 172.21.15.40:0/3487307182 shutdown_connections 2023-09-30T00:48:45.549 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.544+0000 7fa751129700 1 -- 172.21.15.40:0/3487307182 wait complete. 2023-09-30T00:48:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.545+0000 7fa751129700 10 monclient: build_initial_monmap 2023-09-30T00:48:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.545+0000 7fa751129700 10 monclient: monmap: 2023-09-30T00:48:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:45.551 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:45.552 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:45.552 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:45.552 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:45.552 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:45.552 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:45.552 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:45.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.545+0000 7fa751129700 1 librados: starting msgr at 2023-09-30T00:48:45.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.545+0000 7fa751129700 1 librados: starting objecter 2023-09-30T00:48:45.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.546+0000 7fa751129700 1 Processor -- start 2023-09-30T00:48:45.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.546+0000 7fa751129700 1 -- start start 2023-09-30T00:48:45.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.546+0000 7fa751129700 1 librados: setting wanted keys 2023-09-30T00:48:45.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.546+0000 7fa751129700 1 librados: calling monclient init 2023-09-30T00:48:45.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.546+0000 7fa751129700 10 monclient: init 2023-09-30T00:48:45.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:45.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:48:45.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa74c0b1460 0x7fa74c0a9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:45.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 10 monclient(hunting): picked mon.noname-b con 0x7fa74c0b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:45.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa74c0b1830 0x7fa74c13df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:45.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 10 monclient(hunting): picked mon.noname-c con 0x7fa74c0b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:45.553 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa74c13ac20 0x7fa74c0a9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:45.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 10 monclient(hunting): picked mon.noname-a con 0x7fa74c13ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:45.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:45.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c13e600 con 0x7fa74c13ac20 2023-09-30T00:48:45.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:45.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c0ab520 con 0x7fa74c0b1460 2023-09-30T00:48:45.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:45.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c13ed70 con 0x7fa74c0b1830 2023-09-30T00:48:45.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:45.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa751129700 10 monclient(hunting): authenticate will time out at 5386.848134s 2023-09-30T00:48:45.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa74a59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa74c0b1830 0x7fa74c13df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:45.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.547+0000 7fa74ad9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa74c0b1460 0x7fa74c0a9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:45.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74a59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa74c0b1830 0x7fa74c13df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:42564/0 (socket says 172.21.15.40:42564) 2023-09-30T00:48:45.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74a59c700 1 -- 172.21.15.40:0/2726636085 learned_addr learned my addr 172.21.15.40:0/2726636085 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:45.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74a59c700 10 monclient(hunting): get_auth_request con 0x7fa74c0b1830 auth_method 0 2023-09-30T00:48:45.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:45.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:45.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:45.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74ad9d700 10 monclient(hunting): get_auth_request con 0x7fa74c0b1460 auth_method 0 2023-09-30T00:48:45.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:45.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:45.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:45.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:45.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:45.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:45.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:45.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.548+0000 7fa74ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:45.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.549+0000 7fa74ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:45.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.552+0000 7fa74a59c700 10 monclient(hunting): handle_auth_done global_id 4376 payload 824 2023-09-30T00:48:45.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.552+0000 7fa74a59c700 1 -- 172.21.15.40:0/2726636085 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa74c0b1460 msgr2=0x7fa74c0a9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:45.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.552+0000 7fa74a59c700 1 --2- 172.21.15.40:0/2726636085 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa74c0b1460 0x7fa74c0a9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:45.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.552+0000 7fa74a59c700 1 -- 172.21.15.40:0/2726636085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa74c13ac20 msgr2=0x7fa74c0a9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:45.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.552+0000 7fa74a59c700 1 --2- 172.21.15.40:0/2726636085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa74c13ac20 0x7fa74c0a9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:45.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.552+0000 7fa74a59c700 10 monclient: _finish_hunting 0 2023-09-30T00:48:45.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.552+0000 7fa74a59c700 1 monclient: found mon.noname-c 2023-09-30T00:48:45.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.552+0000 7fa74a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:45.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.552+0000 7fa74a59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:48:45.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.552+0000 7fa74a59c700 1 -- 172.21.15.40:0/2726636085 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa744009a20 con 0x7fa74c0b1830 2023-09-30T00:48:45.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.554+0000 7fa74a59c700 1 --2- 172.21.15.40:0/2726636085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa74c0b1830 0x7fa74c13df30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa74000b810 tx=0x7fa740009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:45.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.554+0000 7fa732ffd700 10 client.?.objecter ms_handle_connect 0x7fa74c0b1830 2023-09-30T00:48:45.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.554+0000 7fa732ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:45.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.556+0000 7fa732ffd700 1 -- 172.21.15.40:0/2726636085 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74000f510 con 0x7fa74c0b1830 2023-09-30T00:48:45.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.556+0000 7fa732ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:45.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.557+0000 7fa732ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:45.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.557+0000 7fa732ffd700 10 monclient: dump: 2023-09-30T00:48:45.559 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:45.559 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:45.559 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:45.559 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:45.559 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:45.560 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.557+0000 7fa732ffd700 10 monclient: _finish_auth 0 2023-09-30T00:48:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.557+0000 7fa732ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.557+0000 7fa751129700 5 monclient: authenticate success, global_id 4376 2023-09-30T00:48:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.557+0000 7fa751129700 10 monclient: _renew_subs 2023-09-30T00:48:45.560 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.557+0000 7fa751129700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:48:45.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.557+0000 7fa751129700 1 -- 172.21.15.40:0/2726636085 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74c0a7980 con 0x7fa74c0b1830 2023-09-30T00:48:45.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.557+0000 7fa751129700 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:45.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.557+0000 7fa751129700 10 monclient: _renew_subs 2023-09-30T00:48:45.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.557+0000 7fa751129700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:48:45.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.557+0000 7fa751129700 1 -- 172.21.15.40:0/2726636085 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74c0aa260 con 0x7fa74c0b1830 2023-09-30T00:48:45.561 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.557+0000 7fa751129700 1 librados: init done 2023-09-30T00:48:45.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.558+0000 7fa751129700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:45.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.558+0000 7fa751129700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:45.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.558+0000 7fa751129700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:48:45.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.558+0000 7fa751129700 1 -- 172.21.15.40:0/2726636085 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa74c13e470 con 0x7fa74c0b1830 2023-09-30T00:48:45.562 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.558+0000 7fa732ffd700 1 -- 172.21.15.40:0/2726636085 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74000f880 con 0x7fa74c0b1830 2023-09-30T00:48:45.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.558+0000 7fa732ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:45.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.558+0000 7fa732ffd700 1 -- 172.21.15.40:0/2726636085 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74000fa10 con 0x7fa74c0b1830 2023-09-30T00:48:45.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.558+0000 7fa732ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:45.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.558+0000 7fa732ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:45.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.558+0000 7fa732ffd700 10 monclient: dump: 2023-09-30T00:48:45.563 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:45.563 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:45.563 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:45.563 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:45.563 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:45.564 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:45.564 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:45.564 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:45.564 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:45.564 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:45.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.558+0000 7fa74959a700 4 set_mon_vals no callback set 2023-09-30T00:48:45.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.858+0000 7fa732ffd700 1 -- 172.21.15.40:0/2726636085 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa74000fb70 con 0x7fa74c0b1830 2023-09-30T00:48:45.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.859+0000 7fa732ffd700 1 --2- 172.21.15.40:0/2726636085 >> v2:172.21.15.135:6800/116426 conn(0x7fa738068910 0x7fa73806adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:45.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.859+0000 7fa732ffd700 1 -- 172.21.15.40:0/2726636085 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa740084350 con 0x7fa74c0b1830 2023-09-30T00:48:45.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.859+0000 7fa732ffd700 10 client.4376.objecter ms_dispatch 0x7fa74c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.859+0000 7fa732ffd700 3 client.4376.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.859+0000 7fa732ffd700 3 client.4376.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.859+0000 7fa732ffd700 20 client.4376.objecter dump_active .. 0 homeless 2023-09-30T00:48:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.859+0000 7fa732ffd700 1 -- 172.21.15.40:0/2726636085 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa7400847e0 con 0x7fa74c0b1830 2023-09-30T00:48:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.859+0000 7fa732ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.859+0000 7fa732ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.859+0000 7fa74ad9d700 1 --2- 172.21.15.40:0/2726636085 >> v2:172.21.15.135:6800/116426 conn(0x7fa738068910 0x7fa73806adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.859+0000 7fa74ad9d700 10 monclient: get_auth_request con 0x7fa738068910 auth_method 0 2023-09-30T00:48:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.860+0000 7fa732ffd700 10 client.4376.objecter ms_handle_connect 0x7fa738068910 2023-09-30T00:48:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:45.860+0000 7fa74ad9d700 1 --2- 172.21.15.40:0/2726636085 >> v2:172.21.15.135:6800/116426 conn(0x7fa738068910 0x7fa73806adc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa73c007130 tx=0x7fa73c007eb0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:46.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.118+0000 7fa751129700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:46.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.118+0000 7fa751129700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:46.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.118+0000 7fa751129700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:48:46.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.118+0000 7fa751129700 1 -- 172.21.15.40:0/2726636085 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa74c064a00 con 0x7fa74c0b1830 2023-09-30T00:48:46.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.134+0000 7fa732ffd700 1 -- 172.21.15.40:0/2726636085 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa740051c70 con 0x7fa74c0b1830 2023-09-30T00:48:46.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.134+0000 7fa732ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:46.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.134+0000 7fa732ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:46.135 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:46.135 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:46.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.136+0000 7fa751129700 10 librados: watch_flush enter 2023-09-30T00:48:46.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 10 librados: watch_flush exit 2023-09-30T00:48:46.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 20 client.4376.objecter shutdown clearing up homeless session... 2023-09-30T00:48:46.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 10 client.4376.objecter successfully canceled tick 2023-09-30T00:48:46.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 1 -- 172.21.15.40:0/2726636085 >> v2:172.21.15.135:6800/116426 conn(0x7fa738068910 msgr2=0x7fa73806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:46.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 1 --2- 172.21.15.40:0/2726636085 >> v2:172.21.15.135:6800/116426 conn(0x7fa738068910 0x7fa73806adc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa73c007130 tx=0x7fa73c007eb0 comp rx=0 tx=0).stop 2023-09-30T00:48:46.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 10 monclient: shutdown 2023-09-30T00:48:46.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:46.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 1 -- 172.21.15.40:0/2726636085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa74c0b1830 msgr2=0x7fa74c13df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:46.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 1 --2- 172.21.15.40:0/2726636085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa74c0b1830 0x7fa74c13df30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa74000b810 tx=0x7fa740009250 comp rx=0 tx=0).stop 2023-09-30T00:48:46.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 1 -- 172.21.15.40:0/2726636085 shutdown_connections 2023-09-30T00:48:46.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 1 --2- 172.21.15.40:0/2726636085 >> v2:172.21.15.135:6800/116426 conn(0x7fa738068910 0x7fa73806adc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:46.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 1 --2- 172.21.15.40:0/2726636085 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa74c0b1460 0x7fa74c0a9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:46.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 1 --2- 172.21.15.40:0/2726636085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa74c13ac20 0x7fa74c0a9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:46.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 1 --2- 172.21.15.40:0/2726636085 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa74c0b1830 0x7fa74c13df30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:46.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 1 -- 172.21.15.40:0/2726636085 >> 172.21.15.40:0/2726636085 conn(0x7fa74c0631f0 msgr2=0x7fa74c0afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:46.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 1 -- 172.21.15.40:0/2726636085 shutdown_connections 2023-09-30T00:48:46.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 1 -- 172.21.15.40:0/2726636085 wait complete. 2023-09-30T00:48:46.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:46.137+0000 7fa751129700 1 librados: shutdown 2023-09-30T00:48:47.849 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:02:23.560 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:47.912 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:02:23.623 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:47.929 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:02:23.641 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:47.934 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:02:23.646 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:47.948 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:02:23.660 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:47.962 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:02:23.674 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:47.983 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:23.670 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:47.988 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:23.674 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:48.012 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:23.699 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:48:49.152 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:49.287 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.286+0000 7fbbaa747700 10 monclient: get_monmap_and_config 2023-09-30T00:48:49.287 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.286+0000 7fbbaa747700 10 monclient: build_initial_monmap 2023-09-30T00:48:49.287 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.286+0000 7fbbaa747700 10 monclient: monmap: 2023-09-30T00:48:49.287 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:49.287 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:49.288 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:49.287645+0000 2023-09-30T00:48:49.288 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:49.287645+0000 2023-09-30T00:48:49.288 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:49.288 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:49.288 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:49.288 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:49.288 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:49.288 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:49.288 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:49.288 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:49.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.287+0000 7fbbaa747700 1 Processor -- start 2023-09-30T00:48:49.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.287+0000 7fbbaa747700 1 -- start start 2023-09-30T00:48:49.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.287+0000 7fbbaa747700 10 monclient: init 2023-09-30T00:48:49.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.288+0000 7fbbaa747700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:49.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.288+0000 7fbbaa747700 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-09-30T00:48:49.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.288+0000 7fbbaa747700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbba40b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:49.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.288+0000 7fbbaa747700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbba40b1830 0x7fbba40a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:49.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.288+0000 7fbbaa747700 10 monclient(hunting): picked mon.noname-c con 0x7fbba40b1830 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:49.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.288+0000 7fbbaa747700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbba413ac20 addr v1:172.21.15.40:6789/0 2023-09-30T00:48:49.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.288+0000 7fbbaa747700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba40a5b20 con 0x7fbba413ac20 2023-09-30T00:48:49.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.288+0000 7fbbaa747700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba40a71f0 con 0x7fbba40b1460 2023-09-30T00:48:49.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.288+0000 7fbbaa747700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:49.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.288+0000 7fbbaa747700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba4141e40 con 0x7fbba40b1830 2023-09-30T00:48:49.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.288+0000 7fbbaa747700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:49.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.288+0000 7fbbaa747700 10 monclient(hunting): authenticate will time out at 5390.589134s 2023-09-30T00:48:49.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.288+0000 7fbba8ce4700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbba413ac20 0x7fbba413b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57872/0 (socket says 172.21.15.40:57872) 2023-09-30T00:48:49.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.288+0000 7fbba8ce4700 1 -- 172.21.15.40:0/274119616 learned_addr learned my addr 172.21.15.40:0/274119616 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:48:49.292 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.289+0000 7fbba37fe700 1 --2- 172.21.15.40:0/274119616 >> v2:172.21.15.135:3300/0 conn(0x7fbba40b1830 0x7fbba40a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:49.292 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.289+0000 7fbba37fe700 10 monclient(hunting): get_auth_request con 0x7fbba40b1830 auth_method 0 2023-09-30T00:48:49.292 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.289+0000 7fbba37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:49.292 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.289+0000 7fbba37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:49.292 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.289+0000 7fbba37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:49.292 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.290+0000 7fbba37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:49.293 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.290+0000 7fbba37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:49.293 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.290+0000 7fbba37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:49.293 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.292+0000 7fbba2ffd700 1 -- 172.21.15.40:0/274119616 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569903287 0 0) 0x7fbba40a5b20 con 0x7fbba413ac20 2023-09-30T00:48:49.293 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.292+0000 7fbba2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:49.293 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.292+0000 7fbba2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:49.293 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.292+0000 7fbba2ffd700 10 monclient(hunting): my global_id is 4431 2023-09-30T00:48:49.293 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.292+0000 7fbba2ffd700 1 -- 172.21.15.40:0/274119616 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb8c003570 con 0x7fbba413ac20 2023-09-30T00:48:49.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.293+0000 7fbba37fe700 10 monclient(hunting): handle_auth_done global_id 4471 payload 274 2023-09-30T00:48:49.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.293+0000 7fbba37fe700 1 -- 172.21.15.40:0/274119616 >> v1:172.21.15.135:6789/0 conn(0x7fbba40b1460 legacy=0x7fbba4141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:49.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.293+0000 7fbba37fe700 1 -- 172.21.15.40:0/274119616 >> v1:172.21.15.40:6789/0 conn(0x7fbba413ac20 legacy=0x7fbba413b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:49.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.293+0000 7fbba37fe700 10 monclient: _finish_hunting 0 2023-09-30T00:48:49.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.293+0000 7fbba37fe700 1 monclient: found mon.noname-c 2023-09-30T00:48:49.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.293+0000 7fbba37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:49.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.293+0000 7fbba37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:48:49.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.293+0000 7fbba37fe700 1 -- 172.21.15.40:0/274119616 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba40a74d0 con 0x7fbba40b1830 2023-09-30T00:48:49.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.296+0000 7fbba37fe700 1 --2- 172.21.15.40:0/274119616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbba40b1830 0x7fbba40a5fb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbb98009910 tx=0x7fbb980091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec80471e017cad66 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:49.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbba2ffd700 1 -- 172.21.15.40:0/274119616 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9800b640 con 0x7fbba40b1830 2023-09-30T00:48:49.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbba2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:49.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbba2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:49.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbba2ffd700 10 monclient: dump: 2023-09-30T00:48:49.311 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:49.311 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:49.312 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:49.312 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:49.312 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:49.312 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:49.312 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:49.312 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:49.312 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:49.312 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:49.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbba2ffd700 10 monclient: _finish_auth 0 2023-09-30T00:48:49.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbba2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:49.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbba2ffd700 1 -- 172.21.15.40:0/274119616 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb9800b7a0 con 0x7fbba40b1830 2023-09-30T00:48:49.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbbaa747700 5 monclient: authenticate success, global_id 4471 2023-09-30T00:48:49.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbbaa747700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:49.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbba2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:49.313 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbbaa747700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:49.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbba2ffd700 1 -- 172.21.15.40:0/274119616 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9800bb60 con 0x7fbba40b1830 2023-09-30T00:48:49.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbbaa747700 4 set_mon_vals no callback set 2023-09-30T00:48:49.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbba2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:49.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbba2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:49.314 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbba2ffd700 10 monclient: dump: 2023-09-30T00:48:49.314 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:49.314 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:49.315 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:49.315 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:49.315 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:49.315 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:49.315 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:49.315 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:49.315 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:49.315 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:49.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbbaa747700 10 monclient: shutdown 2023-09-30T00:48:49.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbbaa747700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:49.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbbaa747700 1 -- 172.21.15.40:0/274119616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbba40b1830 msgr2=0x7fbba40a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:49.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.304+0000 7fbbaa747700 1 --2- 172.21.15.40:0/274119616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbba40b1830 0x7fbba40a5fb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbb98009910 tx=0x7fbb980091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:49.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.305+0000 7fbbaa747700 1 -- 172.21.15.40:0/274119616 shutdown_connections 2023-09-30T00:48:49.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.305+0000 7fbbaa747700 1 --2- 172.21.15.40:0/274119616 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbba40b1830 0x7fbba40a5fb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:49.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.305+0000 7fbbaa747700 1 -- 172.21.15.40:0/274119616 >> 172.21.15.40:0/274119616 conn(0x7fbba40631f0 msgr2=0x7fbba40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:49.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.305+0000 7fbbaa747700 1 -- 172.21.15.40:0/274119616 shutdown_connections 2023-09-30T00:48:49.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.305+0000 7fbbaa747700 1 -- 172.21.15.40:0/274119616 wait complete. 2023-09-30T00:48:49.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.305+0000 7fbbaa747700 10 monclient: build_initial_monmap 2023-09-30T00:48:49.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.305+0000 7fbbaa747700 10 monclient: monmap: 2023-09-30T00:48:49.317 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:49.317 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:49.318 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:49.318 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:49.318 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:49.318 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:49.318 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:49.318 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:49.318 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:49.319 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:49.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.306+0000 7fbbaa747700 1 librados: starting msgr at 2023-09-30T00:48:49.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.306+0000 7fbbaa747700 1 librados: starting objecter 2023-09-30T00:48:49.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.306+0000 7fbbaa747700 1 Processor -- start 2023-09-30T00:48:49.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.306+0000 7fbbaa747700 1 -- start start 2023-09-30T00:48:49.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.306+0000 7fbbaa747700 1 librados: setting wanted keys 2023-09-30T00:48:49.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.306+0000 7fbbaa747700 1 librados: calling monclient init 2023-09-30T00:48:49.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.306+0000 7fbbaa747700 10 monclient: init 2023-09-30T00:48:49.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:49.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:48:49.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbba40b1460 0x7fbba414df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:49.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 10 monclient(hunting): picked mon.noname-c con 0x7fbba40b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:49.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbba40b1830 0x7fbba40a8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:49.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 10 monclient(hunting): picked mon.noname-b con 0x7fbba40b1830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:49.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba413ac20 0x7fbba4137c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:49.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 10 monclient(hunting): picked mon.noname-a con 0x7fbba413ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:49.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:49.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba40aac70 con 0x7fbba413ac20 2023-09-30T00:48:49.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:49.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba40a8f00 con 0x7fbba40b1830 2023-09-30T00:48:49.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:49.321 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba413d6d0 con 0x7fbba40b1460 2023-09-30T00:48:49.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:49.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbbaa747700 10 monclient(hunting): authenticate will time out at 5390.608162s 2023-09-30T00:48:49.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbba8ce4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba413ac20 0x7fbba4137c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:49.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.307+0000 7fbba3fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbba40b1460 0x7fbba414df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:49.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.308+0000 7fbba8ce4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba413ac20 0x7fbba4137c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44114/0 (socket says 172.21.15.40:44114) 2023-09-30T00:48:49.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.308+0000 7fbba8ce4700 1 -- 172.21.15.40:0/730576598 learned_addr learned my addr 172.21.15.40:0/730576598 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:49.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.308+0000 7fbba8ce4700 10 monclient(hunting): get_auth_request con 0x7fbba413ac20 auth_method 0 2023-09-30T00:48:49.322 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.308+0000 7fbba8ce4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.308+0000 7fbba8ce4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.308+0000 7fbba8ce4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.308+0000 7fbba3fff700 10 monclient(hunting): get_auth_request con 0x7fbba40b1460 auth_method 0 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.308+0000 7fbba3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.308+0000 7fbba3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.308+0000 7fbba3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.308+0000 7fbba37fe700 1 --2- 172.21.15.40:0/730576598 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbba40b1830 0x7fbba40a8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:49.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.309+0000 7fbba8ce4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:49.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.309+0000 7fbba8ce4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:49.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.309+0000 7fbba8ce4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:49.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.309+0000 7fbba37fe700 10 monclient(hunting): get_auth_request con 0x7fbba40b1830 auth_method 0 2023-09-30T00:48:49.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.309+0000 7fbba37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:49.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.309+0000 7fbba37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:49.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.309+0000 7fbba37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:49.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.309+0000 7fbba3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:49.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.309+0000 7fbba3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:49.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.309+0000 7fbba3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:49.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.309+0000 7fbba37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:49.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.309+0000 7fbba37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:49.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.309+0000 7fbba37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:49.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.313+0000 7fbba37fe700 10 monclient(hunting): handle_auth_done global_id 4477 payload 824 2023-09-30T00:48:49.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.313+0000 7fbba37fe700 1 -- 172.21.15.40:0/730576598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbba40b1460 msgr2=0x7fbba414df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:49.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.313+0000 7fbba37fe700 1 --2- 172.21.15.40:0/730576598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbba40b1460 0x7fbba414df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:49.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.313+0000 7fbba37fe700 1 -- 172.21.15.40:0/730576598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba413ac20 msgr2=0x7fbba4137c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:49.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.313+0000 7fbba37fe700 1 --2- 172.21.15.40:0/730576598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba413ac20 0x7fbba4137c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:49.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.313+0000 7fbba37fe700 10 monclient: _finish_hunting 0 2023-09-30T00:48:49.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.313+0000 7fbba37fe700 1 monclient: found mon.noname-b 2023-09-30T00:48:49.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.313+0000 7fbba37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:49.329 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.313+0000 7fbba37fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:49.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.313+0000 7fbba37fe700 1 -- 172.21.15.40:0/730576598 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb98009420 con 0x7fbba40b1830 2023-09-30T00:48:49.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.315+0000 7fbba37fe700 1 --2- 172.21.15.40:0/730576598 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbba40b1830 0x7fbba40a8810 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbb98016000 tx=0x7fbb98017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:49.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.315+0000 7fbb8bfff700 10 client.?.objecter ms_handle_connect 0x7fbba40b1830 2023-09-30T00:48:49.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.315+0000 7fbb8bfff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:49.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbb8bfff700 1 -- 172.21.15.40:0/730576598 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9801f070 con 0x7fbba40b1830 2023-09-30T00:48:49.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbb8bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:49.330 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbb8bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:49.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbb8bfff700 10 monclient: dump: 2023-09-30T00:48:49.331 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:49.331 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:49.331 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:49.331 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:49.331 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:49.331 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:49.331 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:49.331 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:49.332 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:49.332 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:49.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbb8bfff700 10 monclient: _finish_auth 0 2023-09-30T00:48:49.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbb8bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:49.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbb8bfff700 1 -- 172.21.15.40:0/730576598 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb9800b910 con 0x7fbba40b1830 2023-09-30T00:48:49.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbb8bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:49.332 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbb8bfff700 1 -- 172.21.15.40:0/730576598 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb98011420 con 0x7fbba40b1830 2023-09-30T00:48:49.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbb8bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:49.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbb8bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:49.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbb8bfff700 10 monclient: dump: 2023-09-30T00:48:49.333 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:49.333 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:49.333 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:49.333 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:49.334 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:49.334 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:49.334 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:49.334 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:49.334 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:49.334 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:49.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbbaa747700 5 monclient: authenticate success, global_id 4477 2023-09-30T00:48:49.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbbaa747700 10 monclient: _renew_subs 2023-09-30T00:48:49.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbbaa747700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:49.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbbaa747700 1 -- 172.21.15.40:0/730576598 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba414e9c0 con 0x7fbba40b1830 2023-09-30T00:48:49.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbba27fc700 4 set_mon_vals no callback set 2023-09-30T00:48:49.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbbaa747700 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:49.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbbaa747700 10 monclient: _renew_subs 2023-09-30T00:48:49.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbbaa747700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:49.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbbaa747700 1 -- 172.21.15.40:0/730576598 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba41381d0 con 0x7fbba40b1830 2023-09-30T00:48:49.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.319+0000 7fbbaa747700 1 librados: init done 2023-09-30T00:48:49.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.320+0000 7fbbaa747700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:49.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.320+0000 7fbbaa747700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:49.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.320+0000 7fbbaa747700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:49.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.320+0000 7fbbaa747700 1 -- 172.21.15.40:0/730576598 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba4064a00 con 0x7fbba40b1830 2023-09-30T00:48:49.546 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.545+0000 7fbb8bfff700 1 -- 172.21.15.40:0/730576598 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fbb98011790 con 0x7fbba40b1830 2023-09-30T00:48:49.546 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.546+0000 7fbb8bfff700 1 --2- 172.21.15.40:0/730576598 >> v2:172.21.15.135:6800/116426 conn(0x7fbb90068890 0x7fbb9006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:49.547 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.546+0000 7fbba3fff700 1 --2- 172.21.15.40:0/730576598 >> v2:172.21.15.135:6800/116426 conn(0x7fbb90068890 0x7fbb9006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:49.547 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.546+0000 7fbba3fff700 10 monclient: get_auth_request con 0x7fbb90068890 auth_method 0 2023-09-30T00:48:49.547 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.546+0000 7fbba3fff700 1 --2- 172.21.15.40:0/730576598 >> v2:172.21.15.135:6800/116426 conn(0x7fbb90068890 0x7fbb9006ad40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbb94003a80 tx=0x7fbb94007ec0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:49.547 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.546+0000 7fbb8bfff700 10 client.4477.objecter ms_handle_connect 0x7fbb90068890 2023-09-30T00:48:49.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.572+0000 7fbb8bfff700 1 -- 172.21.15.40:0/730576598 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fbb98056b60 con 0x7fbba40b1830 2023-09-30T00:48:49.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.572+0000 7fbb8bfff700 10 client.4477.objecter ms_dispatch 0x7fbba4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:49.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.572+0000 7fbb8bfff700 3 client.4477.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:49.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.572+0000 7fbb8bfff700 3 client.4477.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:49.573 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.573+0000 7fbb8bfff700 20 client.4477.objecter dump_active .. 0 homeless 2023-09-30T00:48:49.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.582+0000 7fbb8bfff700 1 -- 172.21.15.40:0/730576598 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbb98052ea0 con 0x7fbba40b1830 2023-09-30T00:48:49.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.582+0000 7fbb8bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:49.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.582+0000 7fbb8bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:49.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.837+0000 7fbbaa747700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:49.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.837+0000 7fbbaa747700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:49.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.838+0000 7fbbaa747700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:49.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.838+0000 7fbbaa747700 1 -- 172.21.15.40:0/730576598 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbba40b4480 con 0x7fbba40b1830 2023-09-30T00:48:49.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.843+0000 7fbb8bfff700 1 -- 172.21.15.40:0/730576598 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbb980564c0 con 0x7fbba40b1830 2023-09-30T00:48:49.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.843+0000 7fbb8bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.843+0000 7fbb8bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:49.844 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:49.844 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:49.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 10 librados: watch_flush enter 2023-09-30T00:48:49.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 10 librados: watch_flush exit 2023-09-30T00:48:49.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 20 client.4477.objecter shutdown clearing up homeless session... 2023-09-30T00:48:49.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 10 client.4477.objecter successfully canceled tick 2023-09-30T00:48:49.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 1 -- 172.21.15.40:0/730576598 >> v2:172.21.15.135:6800/116426 conn(0x7fbb90068890 msgr2=0x7fbb9006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:49.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 1 --2- 172.21.15.40:0/730576598 >> v2:172.21.15.135:6800/116426 conn(0x7fbb90068890 0x7fbb9006ad40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbb94003a80 tx=0x7fbb94007ec0 comp rx=0 tx=0).stop 2023-09-30T00:48:49.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 10 monclient: shutdown 2023-09-30T00:48:49.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:49.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 1 -- 172.21.15.40:0/730576598 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbba40b1830 msgr2=0x7fbba40a8810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:49.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 1 --2- 172.21.15.40:0/730576598 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbba40b1830 0x7fbba40a8810 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbb98016000 tx=0x7fbb98017000 comp rx=0 tx=0).stop 2023-09-30T00:48:49.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 1 -- 172.21.15.40:0/730576598 shutdown_connections 2023-09-30T00:48:49.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 1 --2- 172.21.15.40:0/730576598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba413ac20 0x7fbba4137c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:49.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 1 --2- 172.21.15.40:0/730576598 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbba40b1460 0x7fbba414df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:49.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 1 --2- 172.21.15.40:0/730576598 >> v2:172.21.15.135:6800/116426 conn(0x7fbb90068890 0x7fbb9006ad40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:49.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 1 --2- 172.21.15.40:0/730576598 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbba40b1830 0x7fbba40a8810 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:49.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.845+0000 7fbbaa747700 1 -- 172.21.15.40:0/730576598 >> 172.21.15.40:0/730576598 conn(0x7fbba40631f0 msgr2=0x7fbba40b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:49.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.846+0000 7fbbaa747700 1 -- 172.21.15.40:0/730576598 shutdown_connections 2023-09-30T00:48:49.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.846+0000 7fbbaa747700 1 -- 172.21.15.40:0/730576598 wait complete. 2023-09-30T00:48:49.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:49.846+0000 7fbbaa747700 1 librados: shutdown 2023-09-30T00:48:52.861 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:52.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.997+0000 7f6fb2dfc700 10 monclient: get_monmap_and_config 2023-09-30T00:48:52.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.997+0000 7f6fb2dfc700 10 monclient: build_initial_monmap 2023-09-30T00:48:52.998 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.997+0000 7f6fb2dfc700 10 monclient: monmap: 2023-09-30T00:48:52.998 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:52.998 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:52.998690+0000 2023-09-30T00:48:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:52.998690+0000 2023-09-30T00:48:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:52.999 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.998+0000 7f6fb2dfc700 1 Processor -- start 2023-09-30T00:48:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.998+0000 7f6fb2dfc700 1 -- start start 2023-09-30T00:48:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.998+0000 7f6fb2dfc700 10 monclient: init 2023-09-30T00:48:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb2dfc700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb2dfc700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-09-30T00:48:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb2dfc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fac0b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:48:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb2dfc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fac0b1830 addr v1:172.21.15.40:6789/0 2023-09-30T00:48:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb2dfc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6fac0a5fb0 0x7f6fac13b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb2dfc700 10 monclient(hunting): picked mon.noname-a con 0x7f6fac0a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb2dfc700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fac0a5b20 con 0x7f6fac0b1830 2023-09-30T00:48:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb2dfc700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fac0a71f0 con 0x7f6fac0b1460 2023-09-30T00:48:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb2dfc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb2dfc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fac141e40 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb2dfc700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb2dfc700 10 monclient(hunting): authenticate will time out at 5394.300244s 2023-09-30T00:48:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb0b98700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f6fac0b1460 0x7f6fac141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:49486/0 (socket says 172.21.15.40:49486) 2023-09-30T00:48:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb1399700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6fac0a5fb0 0x7f6fac13b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:52.999+0000 7f6fb0b98700 1 -- 172.21.15.40:0/503882843 learned_addr learned my addr 172.21.15.40:0/503882843 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:48:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.000+0000 7f6fb1399700 10 monclient(hunting): get_auth_request con 0x7f6fac0a5fb0 auth_method 0 2023-09-30T00:48:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.000+0000 7f6fb1399700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.000+0000 7f6fb1399700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.000+0000 7f6fb1399700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.001+0000 7f6fb1399700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.001+0000 7f6fb1399700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.001+0000 7f6fb1399700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.002+0000 7f6fab7fe700 1 -- 172.21.15.40:0/503882843 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733018523 0 0) 0x7f6fac0a71f0 con 0x7f6fac0b1460 2023-09-30T00:48:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.002+0000 7f6fab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.002+0000 7f6fab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.002+0000 7f6fab7fe700 10 monclient(hunting): my global_id is 4480 2023-09-30T00:48:53.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.002+0000 7f6fab7fe700 1 -- 172.21.15.40:0/503882843 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f9c003570 con 0x7f6fac0b1460 2023-09-30T00:48:53.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.003+0000 7f6fb1399700 10 monclient(hunting): handle_auth_done global_id 4437 payload 274 2023-09-30T00:48:53.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.003+0000 7f6fb1399700 1 -- 172.21.15.40:0/503882843 >> v1:172.21.15.135:6789/0 conn(0x7f6fac0b1460 legacy=0x7f6fac141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:53.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.003+0000 7f6fb1399700 1 -- 172.21.15.40:0/503882843 >> v1:172.21.15.40:6789/0 conn(0x7f6fac0b1830 legacy=0x7f6fac13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:53.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.003+0000 7f6fb1399700 10 monclient: _finish_hunting 0 2023-09-30T00:48:53.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.003+0000 7f6fb1399700 1 monclient: found mon.noname-a 2023-09-30T00:48:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.003+0000 7f6fb1399700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.003+0000 7f6fb1399700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:48:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.003+0000 7f6fb1399700 1 -- 172.21.15.40:0/503882843 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac0a74d0 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.006 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.006+0000 7f6fb1399700 1 --2- 172.21.15.40:0/503882843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fac0a5fb0 0x7f6fac13b330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4009e90 tx=0x7f6fa4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958cd4e64ba9e313 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:53.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fab7fe700 1 -- 172.21.15.40:0/503882843 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa400e070 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:53.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fab7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:53.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fab7fe700 10 monclient: dump: 2023-09-30T00:48:53.013 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:53.013 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:53.013 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:53.013 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:53.013 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:53.013 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:53.013 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:53.013 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:53.013 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:53.013 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:53.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fab7fe700 10 monclient: _finish_auth 0 2023-09-30T00:48:53.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fab7fe700 1 -- 172.21.15.40:0/503882843 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa400bbb0 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fab7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fab7fe700 1 -- 172.21.15.40:0/503882843 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa4012430 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fb2dfc700 5 monclient: authenticate success, global_id 4437 2023-09-30T00:48:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fb2dfc700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fb2dfc700 4 set_mon_vals no callback set 2023-09-30T00:48:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fab7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.011+0000 7f6fab7fe700 10 monclient: dump: 2023-09-30T00:48:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:53.014 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:53.015 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:53.015 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:53.015 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:53.015 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:53.015 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:53.015 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:53.015 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:53.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.012+0000 7f6fb2dfc700 10 monclient: shutdown 2023-09-30T00:48:53.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.012+0000 7f6fb2dfc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:53.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.012+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/503882843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fac0a5fb0 msgr2=0x7f6fac13b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:53.018 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.012+0000 7f6fb2dfc700 1 --2- 172.21.15.40:0/503882843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fac0a5fb0 0x7f6fac13b330 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4009e90 tx=0x7f6fa4009780 comp rx=0 tx=0).stop 2023-09-30T00:48:53.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.012+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/503882843 shutdown_connections 2023-09-30T00:48:53.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.012+0000 7f6fb2dfc700 1 --2- 172.21.15.40:0/503882843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fac0a5fb0 0x7f6fac13b330 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:53.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.012+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/503882843 >> 172.21.15.40:0/503882843 conn(0x7f6fac0631f0 msgr2=0x7f6fac0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:53.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.012+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/503882843 shutdown_connections 2023-09-30T00:48:53.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.012+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/503882843 wait complete. 2023-09-30T00:48:53.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.013+0000 7f6fb2dfc700 10 monclient: build_initial_monmap 2023-09-30T00:48:53.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.013+0000 7f6fb2dfc700 10 monclient: monmap: 2023-09-30T00:48:53.019 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:53.019 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:53.019 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:53.019 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:53.019 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:53.020 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:53.020 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:53.020 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:53.020 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:53.020 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:53.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.013+0000 7f6fb2dfc700 1 librados: starting msgr at 2023-09-30T00:48:53.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.013+0000 7f6fb2dfc700 1 librados: starting objecter 2023-09-30T00:48:53.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.013+0000 7f6fb2dfc700 1 Processor -- start 2023-09-30T00:48:53.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 1 -- start start 2023-09-30T00:48:53.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 1 librados: setting wanted keys 2023-09-30T00:48:53.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 1 librados: calling monclient init 2023-09-30T00:48:53.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 10 monclient: init 2023-09-30T00:48:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:48:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac0a5fb0 0x7f6fac13dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 10 monclient(hunting): picked mon.noname-b con 0x7f6fac0a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fac0b1460 0x7f6fac150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 10 monclient(hunting): picked mon.noname-a con 0x7f6fac0b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fac0b1830 0x7f6fac14d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 10 monclient(hunting): picked mon.noname-c con 0x7f6fac0b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac0aaa10 con 0x7f6fac0b1460 2023-09-30T00:48:53.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac151280 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac151530 con 0x7f6fac0b1830 2023-09-30T00:48:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.014+0000 7f6fb2dfc700 10 monclient(hunting): authenticate will time out at 5394.315707s 2023-09-30T00:48:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.015+0000 7f6fb0b98700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac0a5fb0 0x7f6fac13dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.015+0000 7f6fb1399700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fac0b1830 0x7f6fac14d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.015+0000 7f6fb0b98700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac0a5fb0 0x7f6fac13dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:36140/0 (socket says 172.21.15.40:36140) 2023-09-30T00:48:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.015+0000 7f6fb0b98700 1 -- 172.21.15.40:0/2910033270 learned_addr learned my addr 172.21.15.40:0/2910033270 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.015+0000 7f6fb0b98700 10 monclient(hunting): get_auth_request con 0x7f6fac0a5fb0 auth_method 0 2023-09-30T00:48:53.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.015+0000 7f6fb0b98700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.015+0000 7f6fb0b98700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.015+0000 7f6fb0b98700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.016+0000 7f6fb1399700 10 monclient(hunting): get_auth_request con 0x7f6fac0b1830 auth_method 0 2023-09-30T00:48:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.016+0000 7f6fb1399700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.016+0000 7f6fb1399700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.016+0000 7f6fb1399700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.016+0000 7f6fb0b98700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.016+0000 7f6fb0b98700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.016+0000 7f6fb0b98700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.016+0000 7f6fabfff700 1 --2- 172.21.15.40:0/2910033270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fac0b1460 0x7f6fac150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.016+0000 7f6fb1399700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:53.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.016+0000 7f6fb1399700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.016+0000 7f6fb1399700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.017+0000 7f6fabfff700 10 monclient(hunting): get_auth_request con 0x7f6fac0b1460 auth_method 0 2023-09-30T00:48:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.017+0000 7f6fabfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.017+0000 7f6fabfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.017+0000 7f6fabfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.017+0000 7f6fabfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.017+0000 7f6fabfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:53.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.017+0000 7f6fabfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:53.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.020+0000 7f6fb0b98700 10 monclient(hunting): handle_auth_done global_id 4483 payload 824 2023-09-30T00:48:53.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.020+0000 7f6fb0b98700 1 -- 172.21.15.40:0/2910033270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fac0b1830 msgr2=0x7f6fac14d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:53.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.020+0000 7f6fb0b98700 1 --2- 172.21.15.40:0/2910033270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fac0b1830 0x7f6fac14d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:53.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.020+0000 7f6fb0b98700 1 -- 172.21.15.40:0/2910033270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fac0b1460 msgr2=0x7f6fac150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:53.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.020+0000 7f6fb0b98700 1 --2- 172.21.15.40:0/2910033270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fac0b1460 0x7f6fac150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:53.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.020+0000 7f6fb0b98700 10 monclient: _finish_hunting 0 2023-09-30T00:48:53.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.020+0000 7f6fb0b98700 1 monclient: found mon.noname-b 2023-09-30T00:48:53.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.020+0000 7f6fb0b98700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:53.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.020+0000 7f6fb0b98700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:53.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.020+0000 7f6fb0b98700 1 -- 172.21.15.40:0/2910033270 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa4009a20 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.022+0000 7f6fb0b98700 1 --2- 172.21.15.40:0/2910033270 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac0a5fb0 0x7f6fac13dfa0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c790 tx=0x7f6fa0007df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:53.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.022+0000 7f6fa8ff9700 10 client.?.objecter ms_handle_connect 0x7f6fac0a5fb0 2023-09-30T00:48:53.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.022+0000 7f6fa8ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:53.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fa8ff9700 1 -- 172.21.15.40:0/2910033270 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0005b30 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fa8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:53.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fa8ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:53.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fa8ff9700 10 monclient: dump: 2023-09-30T00:48:53.028 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:53.028 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:53.028 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:53.028 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:53.028 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:53.028 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:53.028 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:53.028 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:53.029 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fa8ff9700 10 monclient: _finish_auth 0 2023-09-30T00:48:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fa8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fa8ff9700 1 -- 172.21.15.40:0/2910033270 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa0005c90 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fb2dfc700 5 monclient: authenticate success, global_id 4483 2023-09-30T00:48:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fb2dfc700 10 monclient: _renew_subs 2023-09-30T00:48:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fb2dfc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/2910033270 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fac14b3b0 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fa8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fa8ff9700 1 -- 172.21.15.40:0/2910033270 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0007320 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fa8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:53.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fa8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:53.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fa8ff9700 10 monclient: dump: 2023-09-30T00:48:53.030 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:53.030 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:53.030 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:53.030 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:53.030 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:53.030 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:53.030 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:53.030 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:53.031 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:53.031 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:53.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6faaffd700 4 set_mon_vals no callback set 2023-09-30T00:48:53.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fb2dfc700 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:53.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fb2dfc700 10 monclient: _renew_subs 2023-09-30T00:48:53.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fb2dfc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:53.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/2910033270 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fac151c00 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.026+0000 7f6fb2dfc700 1 librados: init done 2023-09-30T00:48:53.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.027+0000 7f6fb2dfc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:53.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.027+0000 7f6fb2dfc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:53.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.027+0000 7f6fb2dfc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:53.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.027+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/2910033270 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fac151810 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.277+0000 7f6fa8ff9700 1 -- 172.21.15.40:0/2910033270 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6fa0007690 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.277+0000 7f6fa8ff9700 1 --2- 172.21.15.40:0/2910033270 >> v2:172.21.15.135:6800/116426 conn(0x7f6f900689e0 0x7f6f9006ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:53.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.278+0000 7f6fa8ff9700 1 -- 172.21.15.40:0/2910033270 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6fa0083790 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.278+0000 7f6fa8ff9700 10 client.4483.objecter ms_dispatch 0x7f6fac001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:53.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.278+0000 7f6fa8ff9700 3 client.4483.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:53.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.278+0000 7f6fa8ff9700 3 client.4483.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:53.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.278+0000 7f6fabfff700 1 --2- 172.21.15.40:0/2910033270 >> v2:172.21.15.135:6800/116426 conn(0x7f6f900689e0 0x7f6f9006ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:53.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.278+0000 7f6fa8ff9700 20 client.4483.objecter dump_active .. 0 homeless 2023-09-30T00:48:53.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.278+0000 7f6fabfff700 10 monclient: get_auth_request con 0x7f6f900689e0 auth_method 0 2023-09-30T00:48:53.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.278+0000 7f6fa8ff9700 10 client.4483.objecter ms_handle_connect 0x7f6f900689e0 2023-09-30T00:48:53.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.278+0000 7f6fabfff700 1 --2- 172.21.15.40:0/2910033270 >> v2:172.21.15.135:6800/116426 conn(0x7f6f900689e0 0x7f6f9006ae90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6fac13c070 tx=0x7f6f98009060 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:53.286 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.286+0000 7f6fa8ff9700 1 -- 172.21.15.40:0/2910033270 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6fa00510b0 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.286 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.286+0000 7f6fa8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:53.287 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.286+0000 7f6fa8ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:53.546 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.546+0000 7f6fb2dfc700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:53.547 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.546+0000 7f6fb2dfc700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:53.547 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.546+0000 7f6fb2dfc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:53.547 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.546+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/2910033270 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6fac13eb80 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.551+0000 7f6fa8ff9700 1 -- 172.21.15.40:0/2910033270 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6fa00546d0 con 0x7f6fac0a5fb0 2023-09-30T00:48:53.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.551+0000 7f6fa8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:53.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.551+0000 7f6fa8ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:53.552 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:53.552 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:53.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 10 librados: watch_flush enter 2023-09-30T00:48:53.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 10 librados: watch_flush exit 2023-09-30T00:48:53.555 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 20 client.4483.objecter shutdown clearing up homeless session... 2023-09-30T00:48:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 10 client.4483.objecter successfully canceled tick 2023-09-30T00:48:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/2910033270 >> v2:172.21.15.135:6800/116426 conn(0x7f6f900689e0 msgr2=0x7f6f9006ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 1 --2- 172.21.15.40:0/2910033270 >> v2:172.21.15.135:6800/116426 conn(0x7f6f900689e0 0x7f6f9006ae90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6fac13c070 tx=0x7f6f98009060 comp rx=0 tx=0).stop 2023-09-30T00:48:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 10 monclient: shutdown 2023-09-30T00:48:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/2910033270 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac0a5fb0 msgr2=0x7f6fac13dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 1 --2- 172.21.15.40:0/2910033270 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac0a5fb0 0x7f6fac13dfa0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c790 tx=0x7f6fa0007df0 comp rx=0 tx=0).stop 2023-09-30T00:48:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/2910033270 shutdown_connections 2023-09-30T00:48:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 1 --2- 172.21.15.40:0/2910033270 >> v2:172.21.15.135:6800/116426 conn(0x7f6f900689e0 0x7f6f9006ae90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 1 --2- 172.21.15.40:0/2910033270 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6fac0a5fb0 0x7f6fac13dfa0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 1 --2- 172.21.15.40:0/2910033270 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6fac0b1830 0x7f6fac14d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 1 --2- 172.21.15.40:0/2910033270 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6fac0b1460 0x7f6fac150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/2910033270 >> 172.21.15.40:0/2910033270 conn(0x7f6fac0631f0 msgr2=0x7f6fac0b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/2910033270 shutdown_connections 2023-09-30T00:48:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 1 -- 172.21.15.40:0/2910033270 wait complete. 2023-09-30T00:48:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:53.554+0000 7f6fb2dfc700 1 librados: shutdown 2023-09-30T00:48:56.572 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.709+0000 7f55cc158700 10 monclient: get_monmap_and_config 2023-09-30T00:48:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.709+0000 7f55cc158700 10 monclient: build_initial_monmap 2023-09-30T00:48:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.709+0000 7f55cc158700 10 monclient: monmap: 2023-09-30T00:48:56.710 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:48:56.710900+0000 2023-09-30T00:48:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:48:56.710900+0000 2023-09-30T00:48:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:48:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:48:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:48:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:48:56.711 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:48:56.711 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:56.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.710+0000 7f55cc158700 1 Processor -- start 2023-09-30T00:48:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.710+0000 7f55cc158700 1 -- start start 2023-09-30T00:48:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.710+0000 7f55cc158700 10 monclient: init 2023-09-30T00:48:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-09-30T00:48:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55c40b1460 0x7f55c40b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 10 monclient(hunting): picked mon.noname-a con 0x7f55c40b1460 addr v2:172.21.15.40:3300/0 2023-09-30T00:48:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40a5fb0 0x7f55c4141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 10 monclient(hunting): picked mon.noname-b con 0x7f55c40a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f55c40af410 0x7f55c40af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 10 monclient(hunting): picked mon.noname-c con 0x7f55c40af410 addr v2:172.21.15.135:3300/0 2023-09-30T00:48:56.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55c40a71f0 con 0x7f55c40b1460 2023-09-30T00:48:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55c40a74d0 con 0x7f55c40af410 2023-09-30T00:48:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c40a63c0 con 0x7f55c40a5fb0 2023-09-30T00:48:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.711+0000 7f55cc158700 10 monclient(hunting): authenticate will time out at 5398.012410s 2023-09-30T00:48:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55c9ef4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55c40b1460 0x7f55c40b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55c96f3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40a5fb0 0x7f55c4141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55ca6f5700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f55c40af410 0x7f55c40af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55ca6f5700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f55c40af410 0x7f55c40af890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:36146/0 (socket says 172.21.15.40:36146) 2023-09-30T00:48:56.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55ca6f5700 1 -- 172.21.15.40:0/1986543437 learned_addr learned my addr 172.21.15.40:0/1986543437 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55ca6f5700 10 monclient(hunting): get_auth_request con 0x7f55c40af410 auth_method 0 2023-09-30T00:48:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55ca6f5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55ca6f5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55ca6f5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55c9ef4700 10 monclient(hunting): get_auth_request con 0x7f55c40b1460 auth_method 0 2023-09-30T00:48:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55c9ef4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55c9ef4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55c9ef4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55c96f3700 10 monclient(hunting): get_auth_request con 0x7f55c40a5fb0 auth_method 0 2023-09-30T00:48:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55c96f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55c96f3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:56.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.712+0000 7f55c96f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.713+0000 7f55ca6f5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.713+0000 7f55ca6f5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.713+0000 7f55ca6f5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.713+0000 7f55c9ef4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.713+0000 7f55c9ef4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.713+0000 7f55c9ef4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.713+0000 7f55c96f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.713+0000 7f55c96f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:56.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.713+0000 7f55c96f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.715+0000 7f55c96f3700 10 monclient(hunting): handle_auth_done global_id 4385 payload 274 2023-09-30T00:48:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.715+0000 7f55c96f3700 1 -- 172.21.15.40:0/1986543437 >> v2:172.21.15.135:3300/0 conn(0x7f55c40af410 msgr2=0x7f55c40af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.715+0000 7f55c96f3700 1 --2- 172.21.15.40:0/1986543437 >> v2:172.21.15.135:3300/0 conn(0x7f55c40af410 0x7f55c40af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.715+0000 7f55c96f3700 1 -- 172.21.15.40:0/1986543437 >> v2:172.21.15.40:3300/0 conn(0x7f55c40b1460 msgr2=0x7f55c40b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.715+0000 7f55c96f3700 1 --2- 172.21.15.40:0/1986543437 >> v2:172.21.15.40:3300/0 conn(0x7f55c40b1460 0x7f55c40b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.715+0000 7f55c96f3700 10 monclient: _finish_hunting 0 2023-09-30T00:48:56.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.715+0000 7f55c96f3700 1 monclient: found mon.noname-b 2023-09-30T00:48:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.715+0000 7f55c96f3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.715+0000 7f55c96f3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:48:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.715+0000 7f55c96f3700 1 -- 172.21.15.40:0/1986543437 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c40a56d0 con 0x7f55c40a5fb0 2023-09-30T00:48:56.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.717+0000 7f55c96f3700 1 --2- 172.21.15.40:0/1986543437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40a5fb0 0x7f55c4141730 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f55c40b05f0 tx=0x7f55b40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9eebb9258fe4658b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:56.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55c8ef2700 1 -- 172.21.15.40:0/1986543437 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b4011630 con 0x7f55c40a5fb0 2023-09-30T00:48:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55c8ef2700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55c8ef2700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55c8ef2700 10 monclient: dump: 2023-09-30T00:48:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:56.730 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:56.731 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:56.732 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55c8ef2700 10 monclient: _finish_auth 0 2023-09-30T00:48:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55c8ef2700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55c8ef2700 1 -- 172.21.15.40:0/1986543437 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b40117b0 con 0x7f55c40a5fb0 2023-09-30T00:48:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55c8ef2700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:56.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55c8ef2700 1 -- 172.21.15.40:0/1986543437 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b4011b70 con 0x7f55c40a5fb0 2023-09-30T00:48:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55cc158700 5 monclient: authenticate success, global_id 4385 2023-09-30T00:48:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55cc158700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55cc158700 4 set_mon_vals no callback set 2023-09-30T00:48:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55c8ef2700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55c8ef2700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:56.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.722+0000 7f55c8ef2700 10 monclient: dump: 2023-09-30T00:48:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:56.734 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:56.735 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:56.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.723+0000 7f55cc158700 10 monclient: shutdown 2023-09-30T00:48:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.723+0000 7f55cc158700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.723+0000 7f55cc158700 1 -- 172.21.15.40:0/1986543437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40a5fb0 msgr2=0x7f55c4141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.723+0000 7f55cc158700 1 --2- 172.21.15.40:0/1986543437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40a5fb0 0x7f55c4141730 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f55c40b05f0 tx=0x7f55b40091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.723+0000 7f55cc158700 1 -- 172.21.15.40:0/1986543437 shutdown_connections 2023-09-30T00:48:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.723+0000 7f55cc158700 1 --2- 172.21.15.40:0/1986543437 >> v2:172.21.15.135:3300/0 conn(0x7f55c40af410 0x7f55c40af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:56.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.723+0000 7f55cc158700 1 --2- 172.21.15.40:0/1986543437 >> v2:172.21.15.40:3300/0 conn(0x7f55c40b1460 0x7f55c40b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.723+0000 7f55cc158700 1 --2- 172.21.15.40:0/1986543437 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40a5fb0 0x7f55c4141730 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.723+0000 7f55cc158700 1 -- 172.21.15.40:0/1986543437 >> 172.21.15.40:0/1986543437 conn(0x7f55c40631f0 msgr2=0x7f55c40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.723+0000 7f55cc158700 1 -- 172.21.15.40:0/1986543437 shutdown_connections 2023-09-30T00:48:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.723+0000 7f55cc158700 1 -- 172.21.15.40:0/1986543437 wait complete. 2023-09-30T00:48:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.724+0000 7f55cc158700 10 monclient: build_initial_monmap 2023-09-30T00:48:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.724+0000 7f55cc158700 10 monclient: monmap: 2023-09-30T00:48:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:48:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:48:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:48:56.738 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:48:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:48:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:48:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:48:56.739 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:56.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.725+0000 7f55cc158700 1 librados: starting msgr at 2023-09-30T00:48:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.725+0000 7f55cc158700 1 librados: starting objecter 2023-09-30T00:48:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.725+0000 7f55cc158700 1 Processor -- start 2023-09-30T00:48:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.725+0000 7f55cc158700 1 -- start start 2023-09-30T00:48:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.725+0000 7f55cc158700 1 librados: setting wanted keys 2023-09-30T00:48:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.725+0000 7f55cc158700 1 librados: calling monclient init 2023-09-30T00:48:56.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.725+0000 7f55cc158700 10 monclient: init 2023-09-30T00:48:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:48:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55c40a5fb0 0x7f55c413e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 10 monclient(hunting): picked mon.noname-a con 0x7f55c40a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:48:56.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55c40af410 0x7f55c414e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 10 monclient(hunting): picked mon.noname-b con 0x7f55c40af410 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:48:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40b1460 0x7f55c414e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 10 monclient(hunting): picked mon.noname-c con 0x7f55c40b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:48:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c40aab70 con 0x7f55c40a5fb0 2023-09-30T00:48:56.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c40aa650 con 0x7f55c40af410 2023-09-30T00:48:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c414eac0 con 0x7f55c40b1460 2023-09-30T00:48:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.726+0000 7f55cc158700 10 monclient(hunting): authenticate will time out at 5398.027445s 2023-09-30T00:48:56.743 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c96f3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55c40af410 0x7f55c414e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c9ef4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55c40a5fb0 0x7f55c413e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c96f3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55c40af410 0x7f55c414e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:36154/0 (socket says 172.21.15.40:36154) 2023-09-30T00:48:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c96f3700 1 -- 172.21.15.40:0/408750292 learned_addr learned my addr 172.21.15.40:0/408750292 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:48:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c96f3700 10 monclient(hunting): get_auth_request con 0x7f55c40af410 auth_method 0 2023-09-30T00:48:56.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c96f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c96f3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c96f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c9ef4700 10 monclient(hunting): get_auth_request con 0x7f55c40a5fb0 auth_method 0 2023-09-30T00:48:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c9ef4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c9ef4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:56.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c9ef4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c96f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c96f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.727+0000 7f55c96f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.728+0000 7f55c9ef4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.728+0000 7f55c9ef4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.728+0000 7f55c9ef4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:56.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.729+0000 7f55ca6f5700 1 --2- 172.21.15.40:0/408750292 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40b1460 0x7f55c414e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.729+0000 7f55ca6f5700 10 monclient(hunting): get_auth_request con 0x7f55c40b1460 auth_method 0 2023-09-30T00:48:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.729+0000 7f55ca6f5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:56.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.729+0000 7f55ca6f5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.729+0000 7f55ca6f5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.730+0000 7f55ca6f5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.730+0000 7f55ca6f5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.730+0000 7f55ca6f5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.730+0000 7f55c96f3700 10 monclient(hunting): handle_auth_done global_id 4489 payload 824 2023-09-30T00:48:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.730+0000 7f55c96f3700 1 -- 172.21.15.40:0/408750292 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40b1460 msgr2=0x7f55c414e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.730+0000 7f55c96f3700 1 --2- 172.21.15.40:0/408750292 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40b1460 0x7f55c414e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.730+0000 7f55c96f3700 1 -- 172.21.15.40:0/408750292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55c40a5fb0 msgr2=0x7f55c413e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.730+0000 7f55c96f3700 1 --2- 172.21.15.40:0/408750292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55c40a5fb0 0x7f55c413e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.730+0000 7f55c96f3700 10 monclient: _finish_hunting 0 2023-09-30T00:48:56.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.730+0000 7f55c96f3700 1 monclient: found mon.noname-b 2023-09-30T00:48:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.730+0000 7f55c96f3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.730+0000 7f55c96f3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:48:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.730+0000 7f55c96f3700 1 -- 172.21.15.40:0/408750292 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b4009420 con 0x7f55c40af410 2023-09-30T00:48:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.733+0000 7f55c96f3700 1 --2- 172.21.15.40:0/408750292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55c40af410 0x7f55c414e040 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f55c40a6500 tx=0x7f55b4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.733+0000 7f55b9ffb700 10 client.?.objecter ms_handle_connect 0x7f55c40af410 2023-09-30T00:48:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.733+0000 7f55b9ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:48:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55b9ffb700 1 -- 172.21.15.40:0/408750292 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b401f070 con 0x7f55c40af410 2023-09-30T00:48:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55b9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55b9ffb700 10 monclient: dump: 2023-09-30T00:48:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:56.751 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:56.752 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55b9ffb700 10 monclient: _finish_auth 0 2023-09-30T00:48:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55b9ffb700 1 -- 172.21.15.40:0/408750292 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b4000bc0 con 0x7f55c40af410 2023-09-30T00:48:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55cc158700 5 monclient: authenticate success, global_id 4489 2023-09-30T00:48:56.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55b9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55b9ffb700 1 -- 172.21.15.40:0/408750292 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b4011b50 con 0x7f55c40af410 2023-09-30T00:48:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55cc158700 10 monclient: _renew_subs 2023-09-30T00:48:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55cc158700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55cc158700 1 -- 172.21.15.40:0/408750292 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c40a8ac0 con 0x7f55c40af410 2023-09-30T00:48:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55bbfff700 4 set_mon_vals no callback set 2023-09-30T00:48:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55b9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55b9ffb700 10 monclient: dump: 2023-09-30T00:48:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:48:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:48:56.753 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:48:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:48:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:48:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:48:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:48:56.754 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:48:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55cc158700 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:48:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55cc158700 10 monclient: _renew_subs 2023-09-30T00:48:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55cc158700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:56.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55cc158700 1 -- 172.21.15.40:0/408750292 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c40aa380 con 0x7f55c40af410 2023-09-30T00:48:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.737+0000 7f55cc158700 1 librados: init done 2023-09-30T00:48:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.738+0000 7f55cc158700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.738+0000 7f55cc158700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.738+0000 7f55cc158700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:56.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.738+0000 7f55cc158700 1 -- 172.21.15.40:0/408750292 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c40aa870 con 0x7f55c40af410 2023-09-30T00:48:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.918+0000 7f55b9ffb700 1 -- 172.21.15.40:0/408750292 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f55b4023070 con 0x7f55c40af410 2023-09-30T00:48:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.918+0000 7f55b9ffb700 1 --2- 172.21.15.40:0/408750292 >> v2:172.21.15.135:6800/116426 conn(0x7f55a8068910 0x7f55a806adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.919+0000 7f55b9ffb700 1 -- 172.21.15.40:0/408750292 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f55b40854e0 con 0x7f55c40af410 2023-09-30T00:48:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.919+0000 7f55b9ffb700 10 client.4489.objecter ms_dispatch 0x7f55c4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:48:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.919+0000 7f55b9ffb700 3 client.4489.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:48:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.919+0000 7f55b9ffb700 3 client.4489.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:48:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.919+0000 7f55b9ffb700 20 client.4489.objecter dump_active .. 0 homeless 2023-09-30T00:48:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.919+0000 7f55c9ef4700 1 --2- 172.21.15.40:0/408750292 >> v2:172.21.15.135:6800/116426 conn(0x7f55a8068910 0x7f55a806adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.919+0000 7f55c9ef4700 10 monclient: get_auth_request con 0x7f55a8068910 auth_method 0 2023-09-30T00:48:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.919+0000 7f55c9ef4700 1 --2- 172.21.15.40:0/408750292 >> v2:172.21.15.135:6800/116426 conn(0x7f55a8068910 0x7f55a806adc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009fd0 tx=0x7f55c000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:56.919+0000 7f55b9ffb700 10 client.4489.objecter ms_handle_connect 0x7f55a8068910 2023-09-30T00:48:57.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.000+0000 7f55b9ffb700 1 -- 172.21.15.40:0/408750292 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f55b4052d80 con 0x7f55c40af410 2023-09-30T00:48:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.000+0000 7f55b9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:57.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.000+0000 7f55b9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:57.286 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.285+0000 7f55cc158700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:57.286 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.285+0000 7f55cc158700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:57.286 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.285+0000 7f55cc158700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:48:57.287 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.285+0000 7f55cc158700 1 -- 172.21.15.40:0/408750292 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f55c4064a00 con 0x7f55c40af410 2023-09-30T00:48:57.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.290+0000 7f55b9ffb700 1 -- 172.21.15.40:0/408750292 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f55b4015100 con 0x7f55c40af410 2023-09-30T00:48:57.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.290+0000 7f55b9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:57.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.290+0000 7f55b9ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:57.291 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:48:57.291 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.292+0000 7f55cc158700 10 librados: watch_flush enter 2023-09-30T00:48:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.292+0000 7f55cc158700 10 librados: watch_flush exit 2023-09-30T00:48:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.292+0000 7f55cc158700 20 client.4489.objecter shutdown clearing up homeless session... 2023-09-30T00:48:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.292+0000 7f55cc158700 10 client.4489.objecter successfully canceled tick 2023-09-30T00:48:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 1 -- 172.21.15.40:0/408750292 >> v2:172.21.15.135:6800/116426 conn(0x7f55a8068910 msgr2=0x7f55a806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 1 --2- 172.21.15.40:0/408750292 >> v2:172.21.15.135:6800/116426 conn(0x7f55a8068910 0x7f55a806adc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009fd0 tx=0x7f55c000b040 comp rx=0 tx=0).stop 2023-09-30T00:48:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 10 monclient: shutdown 2023-09-30T00:48:57.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:57.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 1 -- 172.21.15.40:0/408750292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55c40af410 msgr2=0x7f55c414e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:57.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 1 --2- 172.21.15.40:0/408750292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55c40af410 0x7f55c414e040 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f55c40a6500 tx=0x7f55b4017000 comp rx=0 tx=0).stop 2023-09-30T00:48:57.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 1 -- 172.21.15.40:0/408750292 shutdown_connections 2023-09-30T00:48:57.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 1 --2- 172.21.15.40:0/408750292 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55c40b1460 0x7f55c414e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:57.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 1 --2- 172.21.15.40:0/408750292 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55c40a5fb0 0x7f55c413e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:57.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 1 --2- 172.21.15.40:0/408750292 >> v2:172.21.15.135:6800/116426 conn(0x7f55a8068910 0x7f55a806adc0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:57.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 1 --2- 172.21.15.40:0/408750292 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55c40af410 0x7f55c414e040 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:57.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 1 -- 172.21.15.40:0/408750292 >> 172.21.15.40:0/408750292 conn(0x7f55c40631f0 msgr2=0x7f55c4144aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:57.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 1 -- 172.21.15.40:0/408750292 shutdown_connections 2023-09-30T00:48:57.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 1 -- 172.21.15.40:0/408750292 wait complete. 2023-09-30T00:48:57.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:48:57.293+0000 7f55cc158700 1 librados: shutdown 2023-09-30T00:49:00.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.441+0000 7f094db86700 10 monclient: get_monmap_and_config 2023-09-30T00:49:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.441+0000 7f094db86700 10 monclient: build_initial_monmap 2023-09-30T00:49:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.441+0000 7f094db86700 10 monclient: monmap: 2023-09-30T00:49:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:00.442374+0000 2023-09-30T00:49:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:00.442374+0000 2023-09-30T00:49:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:00.442 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:00.443 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:00.443 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:00.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 1 Processor -- start 2023-09-30T00:49:00.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 1 -- start start 2023-09-30T00:49:00.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 10 monclient: init 2023-09-30T00:49:00.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:00.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-09-30T00:49:00.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09480b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:49:00.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f09480b1830 0x7f09480a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:00.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 10 monclient(hunting): picked mon.noname-c con 0x7f09480b1830 addr v2:172.21.15.135:3300/0 2023-09-30T00:49:00.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f094813ac20 0x7f094813b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:00.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 10 monclient(hunting): picked mon.noname-b con 0x7f094813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:00.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09480a5b20 con 0x7f09480b1460 2023-09-30T00:49:00.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:00.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0948141e40 con 0x7f09480b1830 2023-09-30T00:49:00.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09480a64f0 con 0x7f094813ac20 2023-09-30T00:49:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.442+0000 7f094db86700 10 monclient(hunting): authenticate will time out at 5401.743966s 2023-09-30T00:49:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.443+0000 7f0947fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f094813ac20 0x7f094813b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.443+0000 7f0946ffd700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f09480b1830 0x7f09480a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0947fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f094813ac20 0x7f094813b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:52398/0 (socket says 172.21.15.40:52398) 2023-09-30T00:49:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0947fff700 1 -- 172.21.15.40:0/955610298 learned_addr learned my addr 172.21.15.40:0/955610298 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0947fff700 10 monclient(hunting): get_auth_request con 0x7f094813ac20 auth_method 0 2023-09-30T00:49:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0947fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0947fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0947fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0946ffd700 10 monclient(hunting): get_auth_request con 0x7f09480b1830 auth_method 0 2023-09-30T00:49:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0946ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0946ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0946ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0947fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0947fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0947fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0946ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0946ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:00.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.444+0000 7f0946ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:00.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.447+0000 7f0947fff700 10 monclient(hunting): handle_auth_done global_id 4391 payload 274 2023-09-30T00:49:00.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.447+0000 7f0947fff700 1 -- 172.21.15.40:0/955610298 >> v2:172.21.15.135:3300/0 conn(0x7f09480b1830 msgr2=0x7f09480a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:00.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.447+0000 7f0947fff700 1 --2- 172.21.15.40:0/955610298 >> v2:172.21.15.135:3300/0 conn(0x7f09480b1830 0x7f09480a5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:00.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.447+0000 7f0947fff700 1 -- 172.21.15.40:0/955610298 >> v1:172.21.15.135:6789/0 conn(0x7f09480b1460 legacy=0x7f0948141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:00.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.447+0000 7f0947fff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.447+0000 7f0947fff700 1 monclient: found mon.noname-b 2023-09-30T00:49:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.447+0000 7f0947fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.447+0000 7f0947fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:49:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.447+0000 7f0947fff700 1 -- 172.21.15.40:0/955610298 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09480a71f0 con 0x7f094813ac20 2023-09-30T00:49:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.449+0000 7f0947fff700 1 --2- 172.21.15.40:0/955610298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f094813ac20 0x7f094813b080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0940009f20 tx=0x7f0940009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eacd1d4b554bdd4f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f09467fc700 1 -- 172.21.15.40:0/955610298 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094000e070 con 0x7f094813ac20 2023-09-30T00:49:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f09467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f09467fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f09467fc700 10 monclient: dump: 2023-09-30T00:49:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:00.454 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:00.455 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:00.455 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:00.455 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:00.455 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:00.455 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:00.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f09467fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:00.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f09467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:00.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f09467fc700 1 -- 172.21.15.40:0/955610298 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094000b990 con 0x7f094813ac20 2023-09-30T00:49:00.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f09467fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:00.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f09467fc700 1 -- 172.21.15.40:0/955610298 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09400092e0 con 0x7f094813ac20 2023-09-30T00:49:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f09467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f09467fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f09467fc700 10 monclient: dump: 2023-09-30T00:49:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:00.456 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:00.456 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f094db86700 5 monclient: authenticate success, global_id 4391 2023-09-30T00:49:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f094db86700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:00.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.453+0000 7f094db86700 4 set_mon_vals no callback set 2023-09-30T00:49:00.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.454+0000 7f094db86700 10 monclient: shutdown 2023-09-30T00:49:00.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.454+0000 7f094db86700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:00.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.454+0000 7f094db86700 1 -- 172.21.15.40:0/955610298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f094813ac20 msgr2=0x7f094813b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:00.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.454+0000 7f094db86700 1 --2- 172.21.15.40:0/955610298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f094813ac20 0x7f094813b080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0940009f20 tx=0x7f0940009840 comp rx=0 tx=0).stop 2023-09-30T00:49:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.454+0000 7f094db86700 1 -- 172.21.15.40:0/955610298 shutdown_connections 2023-09-30T00:49:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.454+0000 7f094db86700 1 --2- 172.21.15.40:0/955610298 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f094813ac20 0x7f094813b080 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.454+0000 7f094db86700 1 --2- 172.21.15.40:0/955610298 >> v2:172.21.15.135:3300/0 conn(0x7f09480b1830 0x7f09480a5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.454+0000 7f094db86700 1 -- 172.21.15.40:0/955610298 >> 172.21.15.40:0/955610298 conn(0x7f09480631f0 msgr2=0x7f09480635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.454+0000 7f094db86700 1 -- 172.21.15.40:0/955610298 shutdown_connections 2023-09-30T00:49:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.454+0000 7f094db86700 1 -- 172.21.15.40:0/955610298 wait complete. 2023-09-30T00:49:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.455+0000 7f094db86700 10 monclient: build_initial_monmap 2023-09-30T00:49:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.455+0000 7f094db86700 10 monclient: monmap: 2023-09-30T00:49:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:00.461 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:00.462 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.455+0000 7f094db86700 1 librados: starting msgr at 2023-09-30T00:49:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.455+0000 7f094db86700 1 librados: starting objecter 2023-09-30T00:49:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.455+0000 7f094db86700 1 Processor -- start 2023-09-30T00:49:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.455+0000 7f094db86700 1 -- start start 2023-09-30T00:49:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.455+0000 7f094db86700 1 librados: setting wanted keys 2023-09-30T00:49:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.455+0000 7f094db86700 1 librados: calling monclient init 2023-09-30T00:49:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.455+0000 7f094db86700 10 monclient: init 2023-09-30T00:49:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:49:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09480b1460 0x7f094813e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 10 monclient(hunting): picked mon.noname-c con 0x7f09480b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09480b1830 0x7f0948150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 10 monclient(hunting): picked mon.noname-b con 0x7f09480b1830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f094813ac20 0x7f0948151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 10 monclient(hunting): picked mon.noname-a con 0x7f094813ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09480aa7a0 con 0x7f094813ac20 2023-09-30T00:49:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09480ab560 con 0x7f09480b1830 2023-09-30T00:49:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0948151760 con 0x7f09480b1460 2023-09-30T00:49:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f094db86700 10 monclient(hunting): authenticate will time out at 5401.757307s 2023-09-30T00:49:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.456+0000 7f0946ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09480b1830 0x7f0948150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:00.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f0947fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f094813ac20 0x7f0948151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:00.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f0946ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09480b1830 0x7f0948150ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:36166/0 (socket says 172.21.15.40:36166) 2023-09-30T00:49:00.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f0946ffd700 1 -- 172.21.15.40:0/3868741563 learned_addr learned my addr 172.21.15.40:0/3868741563 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:00.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f0946ffd700 10 monclient(hunting): get_auth_request con 0x7f09480b1830 auth_method 0 2023-09-30T00:49:00.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f0946ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:00.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f0946ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:00.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f0946ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:00.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f09477fe700 1 --2- 172.21.15.40:0/3868741563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09480b1460 0x7f094813e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:00.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f0947fff700 10 monclient(hunting): get_auth_request con 0x7f094813ac20 auth_method 0 2023-09-30T00:49:00.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f0947fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:00.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f0947fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:00.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f0947fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:00.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f0946ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:00.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f0946ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:00.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f0946ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:00.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f09477fe700 10 monclient(hunting): get_auth_request con 0x7f09480b1460 auth_method 0 2023-09-30T00:49:00.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f09477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:00.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f09477fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:00.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.457+0000 7f09477fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:00.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.458+0000 7f0947fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:00.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.458+0000 7f0947fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:00.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.458+0000 7f0947fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:00.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.458+0000 7f09477fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:00.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.458+0000 7f09477fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:00.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.458+0000 7f09477fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:00.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.461+0000 7f0946ffd700 10 monclient(hunting): handle_auth_done global_id 4495 payload 824 2023-09-30T00:49:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.461+0000 7f0946ffd700 1 -- 172.21.15.40:0/3868741563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09480b1460 msgr2=0x7f094813e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.461+0000 7f0946ffd700 1 --2- 172.21.15.40:0/3868741563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09480b1460 0x7f094813e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.461+0000 7f0946ffd700 1 -- 172.21.15.40:0/3868741563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f094813ac20 msgr2=0x7f0948151220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.461+0000 7f0946ffd700 1 --2- 172.21.15.40:0/3868741563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f094813ac20 0x7f0948151220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.461+0000 7f0946ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.461+0000 7f0946ffd700 1 monclient: found mon.noname-b 2023-09-30T00:49:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.461+0000 7f0946ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.461+0000 7f0946ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:49:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.461+0000 7f0946ffd700 1 -- 172.21.15.40:0/3868741563 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0940009ae0 con 0x7f09480b1830 2023-09-30T00:49:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.463+0000 7f0946ffd700 1 --2- 172.21.15.40:0/3868741563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09480b1830 0x7f0948150ce0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f093c00ec80 tx=0x7f093c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.463+0000 7f092f7fe700 10 client.?.objecter ms_handle_connect 0x7f09480b1830 2023-09-30T00:49:00.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.463+0000 7f092f7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f092f7fe700 1 -- 172.21.15.40:0/3868741563 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c009690 con 0x7f09480b1830 2023-09-30T00:49:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f092f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f092f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f092f7fe700 10 monclient: dump: 2023-09-30T00:49:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:00.471 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:00.471 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:00.471 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:00.471 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:00.471 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:00.471 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f092f7fe700 10 monclient: _finish_auth 0 2023-09-30T00:49:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f092f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f092f7fe700 1 -- 172.21.15.40:0/3868741563 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093c00c850 con 0x7f09480b1830 2023-09-30T00:49:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f094db86700 5 monclient: authenticate success, global_id 4495 2023-09-30T00:49:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f092f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f092f7fe700 1 -- 172.21.15.40:0/3868741563 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c00cc10 con 0x7f09480b1830 2023-09-30T00:49:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f094db86700 10 monclient: _renew_subs 2023-09-30T00:49:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f094db86700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f094db86700 1 -- 172.21.15.40:0/3868741563 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09480a9680 con 0x7f09480b1830 2023-09-30T00:49:00.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f092f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:00.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f092f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:49:00.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f092f7fe700 10 monclient: dump: 2023-09-30T00:49:00.473 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:00.473 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:00.473 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:00.473 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:00.473 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:00.473 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:00.473 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:00.473 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:00.474 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:00.474 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:00.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.467+0000 7f0945ffb700 4 set_mon_vals no callback set 2023-09-30T00:49:00.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.468+0000 7f094db86700 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:00.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.468+0000 7f094db86700 10 monclient: _renew_subs 2023-09-30T00:49:00.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.468+0000 7f094db86700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:00.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.468+0000 7f094db86700 1 -- 172.21.15.40:0/3868741563 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09480a9cd0 con 0x7f09480b1830 2023-09-30T00:49:00.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.468+0000 7f094db86700 1 librados: init done 2023-09-30T00:49:00.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.469+0000 7f094db86700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:00.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.469+0000 7f094db86700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:00.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.469+0000 7f094db86700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:00.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.469+0000 7f094db86700 1 -- 172.21.15.40:0/3868741563 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0948142160 con 0x7f09480b1830 2023-09-30T00:49:00.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.725+0000 7f092f7fe700 1 -- 172.21.15.40:0/3868741563 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f093c013470 con 0x7f09480b1830 2023-09-30T00:49:00.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.726+0000 7f092f7fe700 1 --2- 172.21.15.40:0/3868741563 >> v2:172.21.15.135:6800/116426 conn(0x7f09340689e0 0x7f093406ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:00.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.726+0000 7f092f7fe700 1 -- 172.21.15.40:0/3868741563 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f093c084830 con 0x7f09480b1830 2023-09-30T00:49:00.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.726+0000 7f092f7fe700 10 client.4495.objecter ms_dispatch 0x7f0948001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:00.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.726+0000 7f092f7fe700 3 client.4495.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:00.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.726+0000 7f092f7fe700 3 client.4495.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:00.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.726+0000 7f092f7fe700 20 client.4495.objecter dump_active .. 0 homeless 2023-09-30T00:49:00.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.726+0000 7f09477fe700 1 --2- 172.21.15.40:0/3868741563 >> v2:172.21.15.135:6800/116426 conn(0x7f09340689e0 0x7f093406ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:00.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.726+0000 7f09477fe700 10 monclient: get_auth_request con 0x7f09340689e0 auth_method 0 2023-09-30T00:49:00.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.726+0000 7f092f7fe700 10 client.4495.objecter ms_handle_connect 0x7f09340689e0 2023-09-30T00:49:00.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.726+0000 7f09477fe700 1 --2- 172.21.15.40:0/3868741563 >> v2:172.21.15.135:6800/116426 conn(0x7f09340689e0 0x7f093406ae90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0938003a30 tx=0x7f0938007ea0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:00.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.734+0000 7f092f7fe700 1 -- 172.21.15.40:0/3868741563 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f093c052150 con 0x7f09480b1830 2023-09-30T00:49:00.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.734+0000 7f092f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:00.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.734+0000 7f092f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:00.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.991+0000 7f094db86700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:00.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.992+0000 7f094db86700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:00.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.992+0000 7f094db86700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:00.993 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.992+0000 7f094db86700 1 -- 172.21.15.40:0/3868741563 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0948064a00 con 0x7f09480b1830 2023-09-30T00:49:00.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.997+0000 7f092f7fe700 1 -- 172.21.15.40:0/3868741563 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f093c055770 con 0x7f09480b1830 2023-09-30T00:49:00.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.997+0000 7f092f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:00.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.997+0000 7f092f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:00.998 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:00.998 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:01.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 10 librados: watch_flush enter 2023-09-30T00:49:01.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 10 librados: watch_flush exit 2023-09-30T00:49:01.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 20 client.4495.objecter shutdown clearing up homeless session... 2023-09-30T00:49:01.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 10 client.4495.objecter successfully canceled tick 2023-09-30T00:49:01.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 1 -- 172.21.15.40:0/3868741563 >> v2:172.21.15.135:6800/116426 conn(0x7f09340689e0 msgr2=0x7f093406ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:01.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 1 --2- 172.21.15.40:0/3868741563 >> v2:172.21.15.135:6800/116426 conn(0x7f09340689e0 0x7f093406ae90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0938003a30 tx=0x7f0938007ea0 comp rx=0 tx=0).stop 2023-09-30T00:49:01.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 10 monclient: shutdown 2023-09-30T00:49:01.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:01.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 1 -- 172.21.15.40:0/3868741563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09480b1830 msgr2=0x7f0948150ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:01.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 1 --2- 172.21.15.40:0/3868741563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09480b1830 0x7f0948150ce0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f093c00ec80 tx=0x7f093c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:01.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 1 -- 172.21.15.40:0/3868741563 shutdown_connections 2023-09-30T00:49:01.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 1 --2- 172.21.15.40:0/3868741563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f094813ac20 0x7f0948151220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:01.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 1 --2- 172.21.15.40:0/3868741563 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09480b1460 0x7f094813e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:01.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 1 --2- 172.21.15.40:0/3868741563 >> v2:172.21.15.135:6800/116426 conn(0x7f09340689e0 0x7f093406ae90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:01.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 1 --2- 172.21.15.40:0/3868741563 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09480b1830 0x7f0948150ce0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:01.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 1 -- 172.21.15.40:0/3868741563 >> 172.21.15.40:0/3868741563 conn(0x7f09480631f0 msgr2=0x7f0948144880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:01.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 1 -- 172.21.15.40:0/3868741563 shutdown_connections 2023-09-30T00:49:01.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 1 -- 172.21.15.40:0/3868741563 wait complete. 2023-09-30T00:49:01.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:00.999+0000 7f094db86700 1 librados: shutdown 2023-09-30T00:49:04.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:04.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.156+0000 7f91e247d700 10 monclient: get_monmap_and_config 2023-09-30T00:49:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.156+0000 7f91e247d700 10 monclient: build_initial_monmap 2023-09-30T00:49:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.156+0000 7f91e247d700 10 monclient: monmap: 2023-09-30T00:49:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:04.157147+0000 2023-09-30T00:49:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:04.157147+0000 2023-09-30T00:49:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:04.158 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:04.158 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:04.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.156+0000 7f91e247d700 1 Processor -- start 2023-09-30T00:49:04.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.157+0000 7f91e247d700 1 -- start start 2023-09-30T00:49:04.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.157+0000 7f91e247d700 10 monclient: init 2023-09-30T00:49:04.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.157+0000 7f91e247d700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:04.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.157+0000 7f91e247d700 10 monclient: _add_conns ranks=[0,1,3,2,4] 2023-09-30T00:49:04.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.157+0000 7f91e247d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91dc0b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:49:04.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.157+0000 7f91e247d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91dc0b1830 addr v1:172.21.15.135:6789/0 2023-09-30T00:49:04.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.157+0000 7f91e247d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f91dc0a5fb0 0x7f91dc13b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:04.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.157+0000 7f91e247d700 10 monclient(hunting): picked mon.noname-a con 0x7f91dc0a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:49:04.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.157+0000 7f91e247d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91dc0a5b20 con 0x7f91dc0b1460 2023-09-30T00:49:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.157+0000 7f91e247d700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91dc0a71f0 con 0x7f91dc0b1830 2023-09-30T00:49:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.157+0000 7f91e247d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.157+0000 7f91e247d700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91dc141e40 con 0x7f91dc0a5fb0 2023-09-30T00:49:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.157+0000 7f91e247d700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.157+0000 7f91e247d700 10 monclient(hunting): authenticate will time out at 5405.458767s 2023-09-30T00:49:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.158+0000 7f91db7fe700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f91dc0b1830 0x7f91dc13ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:47546/0 (socket says 172.21.15.40:47546) 2023-09-30T00:49:04.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.158+0000 7f91db7fe700 1 -- 172.21.15.40:0/2257634412 learned_addr learned my addr 172.21.15.40:0/2257634412 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:49:04.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.192+0000 7f91e0a1a700 1 --2- 172.21.15.40:0/2257634412 >> v2:172.21.15.40:3300/0 conn(0x7f91dc0a5fb0 0x7f91dc13b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:04.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.193+0000 7f91e0a1a700 10 monclient(hunting): get_auth_request con 0x7f91dc0a5fb0 auth_method 0 2023-09-30T00:49:04.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.193+0000 7f91e0a1a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:04.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.193+0000 7f91e0a1a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:04.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.193+0000 7f91e0a1a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:04.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.193+0000 7f91e0a1a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:04.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.193+0000 7f91e0a1a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:04.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.193+0000 7f91e0a1a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:04.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.196+0000 7f91e0a1a700 10 monclient(hunting): handle_auth_done global_id 4452 payload 274 2023-09-30T00:49:04.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.196+0000 7f91e0a1a700 1 -- 172.21.15.40:0/2257634412 >> v1:172.21.15.135:6789/0 conn(0x7f91dc0b1830 legacy=0x7f91dc13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:04.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.196+0000 7f91e0a1a700 1 -- 172.21.15.40:0/2257634412 >> v1:172.21.15.40:6789/0 conn(0x7f91dc0b1460 legacy=0x7f91dc141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:04.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.196+0000 7f91e0a1a700 10 monclient: _finish_hunting 0 2023-09-30T00:49:04.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.196+0000 7f91e0a1a700 1 monclient: found mon.noname-a 2023-09-30T00:49:04.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.196+0000 7f91e0a1a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:04.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.196+0000 7f91e0a1a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:49:04.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.196+0000 7f91e0a1a700 1 -- 172.21.15.40:0/2257634412 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc0a74d0 con 0x7f91dc0a5fb0 2023-09-30T00:49:04.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.199+0000 7f91e0a1a700 1 --2- 172.21.15.40:0/2257634412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91dc0a5fb0 0x7f91dc13b330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009e90 tx=0x7f91d4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347a06811c9a9d6a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:04.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91daffd700 1 -- 172.21.15.40:0/2257634412 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d400e070 con 0x7f91dc0a5fb0 2023-09-30T00:49:04.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:04.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91daffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:04.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91daffd700 10 monclient: dump: 2023-09-30T00:49:04.208 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:04.208 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:04.208 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:04.208 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:04.208 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:04.208 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:04.208 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:04.208 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:04.209 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:04.209 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:04.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91daffd700 10 monclient: _finish_auth 0 2023-09-30T00:49:04.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91daffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:04.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91daffd700 1 -- 172.21.15.40:0/2257634412 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d400bbb0 con 0x7f91dc0a5fb0 2023-09-30T00:49:04.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91daffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:04.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91daffd700 1 -- 172.21.15.40:0/2257634412 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4012430 con 0x7f91dc0a5fb0 2023-09-30T00:49:04.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91daffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:04.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91daffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:04.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91daffd700 10 monclient: dump: 2023-09-30T00:49:04.209 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:04.209 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:04.210 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:04.210 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:04.210 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:04.210 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:04.210 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:04.210 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:04.210 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:04.210 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:04.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91e247d700 5 monclient: authenticate success, global_id 4452 2023-09-30T00:49:04.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91e247d700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:04.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.205+0000 7f91e247d700 4 set_mon_vals no callback set 2023-09-30T00:49:04.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.206+0000 7f91e247d700 10 monclient: shutdown 2023-09-30T00:49:04.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.206+0000 7f91e247d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:04.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.206+0000 7f91e247d700 1 -- 172.21.15.40:0/2257634412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91dc0a5fb0 msgr2=0x7f91dc13b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:04.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.206+0000 7f91e247d700 1 --2- 172.21.15.40:0/2257634412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91dc0a5fb0 0x7f91dc13b330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009e90 tx=0x7f91d4009780 comp rx=0 tx=0).stop 2023-09-30T00:49:04.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.206+0000 7f91e247d700 1 -- 172.21.15.40:0/2257634412 shutdown_connections 2023-09-30T00:49:04.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.206+0000 7f91e247d700 1 --2- 172.21.15.40:0/2257634412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91dc0a5fb0 0x7f91dc13b330 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:04.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.206+0000 7f91e247d700 1 -- 172.21.15.40:0/2257634412 >> 172.21.15.40:0/2257634412 conn(0x7f91dc0631f0 msgr2=0x7f91dc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:04.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.206+0000 7f91e247d700 1 -- 172.21.15.40:0/2257634412 shutdown_connections 2023-09-30T00:49:04.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.206+0000 7f91e247d700 1 -- 172.21.15.40:0/2257634412 wait complete. 2023-09-30T00:49:04.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.207+0000 7f91e247d700 10 monclient: build_initial_monmap 2023-09-30T00:49:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.207+0000 7f91e247d700 10 monclient: monmap: 2023-09-30T00:49:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:04.212 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:04.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.207+0000 7f91e247d700 1 librados: starting msgr at 2023-09-30T00:49:04.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.207+0000 7f91e247d700 1 librados: starting objecter 2023-09-30T00:49:04.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.207+0000 7f91e247d700 1 Processor -- start 2023-09-30T00:49:04.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.207+0000 7f91e247d700 1 -- start start 2023-09-30T00:49:04.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.207+0000 7f91e247d700 1 librados: setting wanted keys 2023-09-30T00:49:04.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.207+0000 7f91e247d700 1 librados: calling monclient init 2023-09-30T00:49:04.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.207+0000 7f91e247d700 10 monclient: init 2023-09-30T00:49:04.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:04.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:49:04.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91dc0a5fb0 0x7f91dc0aa360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:04.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 10 monclient(hunting): picked mon.noname-b con 0x7f91dc0a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:04.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91dc0b1460 0x7f91dc13e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:04.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 10 monclient(hunting): picked mon.noname-c con 0x7f91dc0b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:04.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91dc0b1830 0x7f91dc151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:04.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 10 monclient(hunting): picked mon.noname-a con 0x7f91dc0b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:04.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:04.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0ab560 con 0x7f91dc0b1830 2023-09-30T00:49:04.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:04.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0ab7d0 con 0x7f91dc0a5fb0 2023-09-30T00:49:04.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:04.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0a9e40 con 0x7f91dc0b1460 2023-09-30T00:49:04.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:04.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.208+0000 7f91e247d700 10 monclient(hunting): authenticate will time out at 5405.509673s 2023-09-30T00:49:04.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.209+0000 7f91e0a1a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91dc0b1830 0x7f91dc151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:04.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.210+0000 7f91e0a1a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91dc0b1830 0x7f91dc151060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41654/0 (socket says 172.21.15.40:41654) 2023-09-30T00:49:04.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.210+0000 7f91e0a1a700 1 -- 172.21.15.40:0/2511665424 learned_addr learned my addr 172.21.15.40:0/2511665424 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:04.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.210+0000 7f91e0a1a700 10 monclient(hunting): get_auth_request con 0x7f91dc0b1830 auth_method 0 2023-09-30T00:49:04.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.210+0000 7f91e0a1a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:04.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.210+0000 7f91e0a1a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:04.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.210+0000 7f91e0a1a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:04.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.211+0000 7f91e0a1a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:04.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.211+0000 7f91e0a1a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:04.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.211+0000 7f91e0a1a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:04.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.216+0000 7f91e0a1a700 10 monclient(hunting): handle_auth_done global_id 4455 payload 824 2023-09-30T00:49:04.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.216+0000 7f91e0a1a700 1 -- 172.21.15.40:0/2511665424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91dc0b1460 msgr2=0x7f91dc13e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:04.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.216+0000 7f91e0a1a700 1 --2- 172.21.15.40:0/2511665424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91dc0b1460 0x7f91dc13e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:04.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.216+0000 7f91e0a1a700 1 -- 172.21.15.40:0/2511665424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91dc0a5fb0 msgr2=0x7f91dc0aa360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:04.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.216+0000 7f91e0a1a700 1 --2- 172.21.15.40:0/2511665424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91dc0a5fb0 0x7f91dc0aa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:04.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.216+0000 7f91e0a1a700 10 monclient: _finish_hunting 0 2023-09-30T00:49:04.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.216+0000 7f91e0a1a700 1 monclient: found mon.noname-a 2023-09-30T00:49:04.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.216+0000 7f91e0a1a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:04.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.216+0000 7f91e0a1a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:49:04.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.216+0000 7f91e0a1a700 1 -- 172.21.15.40:0/2511665424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d4009a20 con 0x7f91dc0b1830 2023-09-30T00:49:04.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.220+0000 7f91e0a1a700 1 --2- 172.21.15.40:0/2511665424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91dc0b1830 0x7f91dc151060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f91d40099f0 tx=0x7f91d4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:04.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.220+0000 7f91c3fff700 10 client.?.objecter ms_handle_connect 0x7f91dc0b1830 2023-09-30T00:49:04.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.220+0000 7f91c3fff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:04.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91c3fff700 1 -- 172.21.15.40:0/2511665424 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d401d640 con 0x7f91dc0b1830 2023-09-30T00:49:04.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:04.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91c3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:04.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91c3fff700 10 monclient: dump: 2023-09-30T00:49:04.226 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:04.227 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91c3fff700 10 monclient: _finish_auth 0 2023-09-30T00:49:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91c3fff700 1 -- 172.21.15.40:0/2511665424 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d401d7c0 con 0x7f91dc0b1830 2023-09-30T00:49:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91e247d700 5 monclient: authenticate success, global_id 4455 2023-09-30T00:49:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91e247d700 10 monclient: _renew_subs 2023-09-30T00:49:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91e247d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:49:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91e247d700 1 -- 172.21.15.40:0/2511665424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91dc13ed80 con 0x7f91dc0b1830 2023-09-30T00:49:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91c3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91c3fff700 1 -- 172.21.15.40:0/2511665424 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d401db80 con 0x7f91dc0b1830 2023-09-30T00:49:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91c3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91c3fff700 10 monclient: dump: 2023-09-30T00:49:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:04.230 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:04.230 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:04.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91da7fc700 4 set_mon_vals no callback set 2023-09-30T00:49:04.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91e247d700 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:04.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91e247d700 10 monclient: _renew_subs 2023-09-30T00:49:04.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91e247d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:49:04.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91e247d700 1 -- 172.21.15.40:0/2511665424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91dc0a9880 con 0x7f91dc0b1830 2023-09-30T00:49:04.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.225+0000 7f91e247d700 1 librados: init done 2023-09-30T00:49:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.226+0000 7f91e247d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.226+0000 7f91e247d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.226+0000 7f91e247d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:49:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.226+0000 7f91e247d700 1 -- 172.21.15.40:0/2511665424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91dc0a9c80 con 0x7f91dc0b1830 2023-09-30T00:49:04.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.537+0000 7f91c3fff700 1 -- 172.21.15.40:0/2511665424 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f91d4024070 con 0x7f91dc0b1830 2023-09-30T00:49:04.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.538+0000 7f91c3fff700 1 --2- 172.21.15.40:0/2511665424 >> v2:172.21.15.135:6800/116426 conn(0x7f91c8068960 0x7f91c806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:04.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.538+0000 7f91c3fff700 1 -- 172.21.15.40:0/2511665424 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f91d4086c60 con 0x7f91dc0b1830 2023-09-30T00:49:04.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.538+0000 7f91c3fff700 10 client.4455.objecter ms_dispatch 0x7f91dc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:04.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.538+0000 7f91c3fff700 3 client.4455.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:04.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.538+0000 7f91c3fff700 3 client.4455.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:04.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.538+0000 7f91dbfff700 1 --2- 172.21.15.40:0/2511665424 >> v2:172.21.15.135:6800/116426 conn(0x7f91c8068960 0x7f91c806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:04.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.538+0000 7f91c3fff700 20 client.4455.objecter dump_active .. 0 homeless 2023-09-30T00:49:04.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.538+0000 7f91dbfff700 10 monclient: get_auth_request con 0x7f91c8068960 auth_method 0 2023-09-30T00:49:04.540 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.538+0000 7f91c3fff700 1 -- 172.21.15.40:0/2511665424 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f91d40871e0 con 0x7f91dc0b1830 2023-09-30T00:49:04.540 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.538+0000 7f91c3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:04.540 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.538+0000 7f91c3fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:04.540 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.539+0000 7f91c3fff700 10 client.4455.objecter ms_handle_connect 0x7f91c8068960 2023-09-30T00:49:04.540 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.539+0000 7f91dbfff700 1 --2- 172.21.15.40:0/2511665424 >> v2:172.21.15.135:6800/116426 conn(0x7f91c8068960 0x7f91c806ae10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f91cc0097c0 tx=0x7f91cc006ce0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:04.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.797+0000 7f91e247d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:04.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.797+0000 7f91e247d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:04.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.797+0000 7f91e247d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:49:04.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.797+0000 7f91e247d700 1 -- 172.21.15.40:0/2511665424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f91dc0aa050 con 0x7f91dc0b1830 2023-09-30T00:49:04.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.802+0000 7f91c3fff700 1 -- 172.21.15.40:0/2511665424 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f91d4054680 con 0x7f91dc0b1830 2023-09-30T00:49:04.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.802+0000 7f91c3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:04.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.802+0000 7f91c3fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:04.803 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:04.803 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:04.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.804+0000 7f91e247d700 10 librados: watch_flush enter 2023-09-30T00:49:04.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.804+0000 7f91e247d700 10 librados: watch_flush exit 2023-09-30T00:49:04.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.804+0000 7f91e247d700 20 client.4455.objecter shutdown clearing up homeless session... 2023-09-30T00:49:04.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.804+0000 7f91e247d700 10 client.4455.objecter successfully canceled tick 2023-09-30T00:49:04.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 1 -- 172.21.15.40:0/2511665424 >> v2:172.21.15.135:6800/116426 conn(0x7f91c8068960 msgr2=0x7f91c806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:04.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 1 --2- 172.21.15.40:0/2511665424 >> v2:172.21.15.135:6800/116426 conn(0x7f91c8068960 0x7f91c806ae10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f91cc0097c0 tx=0x7f91cc006ce0 comp rx=0 tx=0).stop 2023-09-30T00:49:04.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 10 monclient: shutdown 2023-09-30T00:49:04.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:04.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 1 -- 172.21.15.40:0/2511665424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91dc0b1830 msgr2=0x7f91dc151060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:04.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 1 --2- 172.21.15.40:0/2511665424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91dc0b1830 0x7f91dc151060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f91d40099f0 tx=0x7f91d4018000 comp rx=0 tx=0).stop 2023-09-30T00:49:04.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 1 -- 172.21.15.40:0/2511665424 shutdown_connections 2023-09-30T00:49:04.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 1 --2- 172.21.15.40:0/2511665424 >> v2:172.21.15.135:6800/116426 conn(0x7f91c8068960 0x7f91c806ae10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:04.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 1 --2- 172.21.15.40:0/2511665424 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91dc0a5fb0 0x7f91dc0aa360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:04.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 1 --2- 172.21.15.40:0/2511665424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91dc0b1830 0x7f91dc151060 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:04.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 1 --2- 172.21.15.40:0/2511665424 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91dc0b1460 0x7f91dc13e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:04.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 1 -- 172.21.15.40:0/2511665424 >> 172.21.15.40:0/2511665424 conn(0x7f91dc0631f0 msgr2=0x7f91dc0b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:04.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 1 -- 172.21.15.40:0/2511665424 shutdown_connections 2023-09-30T00:49:04.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 1 -- 172.21.15.40:0/2511665424 wait complete. 2023-09-30T00:49:04.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:04.805+0000 7f91e247d700 1 librados: shutdown 2023-09-30T00:49:07.819 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:07.941 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:02:43.652 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:07.965 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:02:43.676 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.971+0000 7f35d721a700 10 monclient: get_monmap_and_config 2023-09-30T00:49:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.971+0000 7f35d721a700 10 monclient: build_initial_monmap 2023-09-30T00:49:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.971+0000 7f35d721a700 10 monclient: monmap: 2023-09-30T00:49:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:07.972630+0000 2023-09-30T00:49:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:07.972630+0000 2023-09-30T00:49:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:07.973 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:07.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.972+0000 7f35d721a700 1 Processor -- start 2023-09-30T00:49:07.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.972+0000 7f35d721a700 1 -- start start 2023-09-30T00:49:07.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.972+0000 7f35d721a700 10 monclient: init 2023-09-30T00:49:07.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.973+0000 7f35d721a700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:07.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.973+0000 7f35d721a700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-09-30T00:49:07.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.973+0000 7f35d721a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35d00b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:49:07.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.973+0000 7f35d721a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f35d00b1830 0x7f35d00a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:07.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.973+0000 7f35d721a700 10 monclient(hunting): picked mon.noname-a con 0x7f35d00b1830 addr v2:172.21.15.40:3300/0 2023-09-30T00:49:07.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.973+0000 7f35d721a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35d013ac20 0x7f35d013b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:07.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.973+0000 7f35d721a700 10 monclient(hunting): picked mon.noname-b con 0x7f35d013ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:07.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.973+0000 7f35d721a700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d00a5b20 con 0x7f35d00b1460 2023-09-30T00:49:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.973+0000 7f35d721a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.973+0000 7f35d721a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35d0141e40 con 0x7f35d00b1830 2023-09-30T00:49:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.973+0000 7f35d721a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.973+0000 7f35d721a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00a64f0 con 0x7f35d013ac20 2023-09-30T00:49:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.973+0000 7f35d721a700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.973+0000 7f35d721a700 10 monclient(hunting): authenticate will time out at 5409.274688s 2023-09-30T00:49:07.997 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:43.684 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:08.000 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:43.686 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:08.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.999+0000 7f35d4fb6700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f35d00b1460 0x7f35d0141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:47552/0 (socket says 172.21.15.40:47552) 2023-09-30T00:49:08.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:07.999+0000 7f35d4fb6700 1 -- 172.21.15.40:0/3178611909 learned_addr learned my addr 172.21.15.40:0/3178611909 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:49:08.000 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:02:43.712 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:08.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.001+0000 7f35d57b7700 1 --2- 172.21.15.40:0/3178611909 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35d013ac20 0x7f35d013b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:08.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.002+0000 7f35d57b7700 10 monclient(hunting): get_auth_request con 0x7f35d013ac20 auth_method 0 2023-09-30T00:49:08.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.002+0000 7f35d57b7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:08.002 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.002+0000 7f35d57b7700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:08.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.002+0000 7f35d57b7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:08.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.003+0000 7f35d57b7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:08.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.003+0000 7f35d57b7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:08.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.003+0000 7f35d57b7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:08.004 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:02:43.715 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:08.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.016+0000 7f35d57b7700 10 monclient(hunting): handle_auth_done global_id 4397 payload 274 2023-09-30T00:49:08.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.016+0000 7f35d57b7700 1 -- 172.21.15.40:0/3178611909 >> v2:172.21.15.40:3300/0 conn(0x7f35d00b1830 msgr2=0x7f35d00a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:08.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.016+0000 7f35d57b7700 1 --2- 172.21.15.40:0/3178611909 >> v2:172.21.15.40:3300/0 conn(0x7f35d00b1830 0x7f35d00a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:08.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.016+0000 7f35d57b7700 1 -- 172.21.15.40:0/3178611909 >> v1:172.21.15.135:6789/0 conn(0x7f35d00b1460 legacy=0x7f35d0141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:08.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.016+0000 7f35d57b7700 10 monclient: _finish_hunting 0 2023-09-30T00:49:08.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.016+0000 7f35d57b7700 1 monclient: found mon.noname-b 2023-09-30T00:49:08.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.016+0000 7f35d57b7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:08.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.016+0000 7f35d57b7700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:49:08.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.016+0000 7f35d57b7700 1 -- 172.21.15.40:0/3178611909 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d00a71f0 con 0x7f35d013ac20 2023-09-30T00:49:08.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.019+0000 7f35d57b7700 1 --2- 172.21.15.40:0/3178611909 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35d013ac20 0x7f35d013b080 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f35c8009e60 tx=0x7f35c8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=309f3ade892794fd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:08.030 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:02:43.717 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:08.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.033+0000 7f35cf7fe700 1 -- 172.21.15.40:0/3178611909 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c800e070 con 0x7f35d013ac20 2023-09-30T00:49:08.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.033+0000 7f35cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:08.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.033+0000 7f35cf7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:08.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.033+0000 7f35cf7fe700 10 monclient: dump: 2023-09-30T00:49:08.035 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:08.035 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:08.035 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:08.035 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:08.035 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:08.035 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:08.035 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:08.035 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:08.035 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:08.035 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:08.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35cf7fe700 10 monclient: _finish_auth 0 2023-09-30T00:49:08.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35cf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:08.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35cf7fe700 1 -- 172.21.15.40:0/3178611909 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c800bc30 con 0x7f35d013ac20 2023-09-30T00:49:08.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35d721a700 5 monclient: authenticate success, global_id 4397 2023-09-30T00:49:08.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35d721a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:08.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35cf7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:08.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35d721a700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:08.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35cf7fe700 1 -- 172.21.15.40:0/3178611909 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c8012430 con 0x7f35d013ac20 2023-09-30T00:49:08.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35d721a700 4 set_mon_vals no callback set 2023-09-30T00:49:08.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35cf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:08.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35cf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:08.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35cf7fe700 10 monclient: dump: 2023-09-30T00:49:08.036 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:08.037 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:08.037 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:08.037 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:08.037 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:08.037 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:08.037 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:08.037 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:08.037 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:08.037 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:08.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35d721a700 10 monclient: shutdown 2023-09-30T00:49:08.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35d721a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:08.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35d721a700 1 -- 172.21.15.40:0/3178611909 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35d013ac20 msgr2=0x7f35d013b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:08.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35d721a700 1 --2- 172.21.15.40:0/3178611909 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35d013ac20 0x7f35d013b080 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f35c8009e60 tx=0x7f35c8009780 comp rx=0 tx=0).stop 2023-09-30T00:49:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35d721a700 1 -- 172.21.15.40:0/3178611909 shutdown_connections 2023-09-30T00:49:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35d721a700 1 --2- 172.21.15.40:0/3178611909 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35d013ac20 0x7f35d013b080 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35d721a700 1 --2- 172.21.15.40:0/3178611909 >> v2:172.21.15.40:3300/0 conn(0x7f35d00b1830 0x7f35d00a5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35d721a700 1 -- 172.21.15.40:0/3178611909 >> 172.21.15.40:0/3178611909 conn(0x7f35d00631f0 msgr2=0x7f35d00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35d721a700 1 -- 172.21.15.40:0/3178611909 shutdown_connections 2023-09-30T00:49:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.034+0000 7f35d721a700 1 -- 172.21.15.40:0/3178611909 wait complete. 2023-09-30T00:49:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.035+0000 7f35d721a700 10 monclient: build_initial_monmap 2023-09-30T00:49:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.035+0000 7f35d721a700 10 monclient: monmap: 2023-09-30T00:49:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:08.040 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:08.041 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:08.041 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:08.041 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:08.041 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:08.041 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:08.041 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:08.041 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:08.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.035+0000 7f35d721a700 1 librados: starting msgr at 2023-09-30T00:49:08.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.035+0000 7f35d721a700 1 librados: starting objecter 2023-09-30T00:49:08.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.036+0000 7f35d721a700 1 Processor -- start 2023-09-30T00:49:08.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.036+0000 7f35d721a700 1 -- start start 2023-09-30T00:49:08.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.036+0000 7f35d721a700 1 librados: setting wanted keys 2023-09-30T00:49:08.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.036+0000 7f35d721a700 1 librados: calling monclient init 2023-09-30T00:49:08.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.036+0000 7f35d721a700 10 monclient: init 2023-09-30T00:49:08.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.036+0000 7f35d721a700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:08.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.036+0000 7f35d721a700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:49:08.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.036+0000 7f35d721a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35d00b1460 0x7f35d013e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:08.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.037+0000 7f35d721a700 10 monclient(hunting): picked mon.noname-c con 0x7f35d00b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:08.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.037+0000 7f35d721a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35d00b1830 0x7f35d00a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:08.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.037+0000 7f35d721a700 10 monclient(hunting): picked mon.noname-a con 0x7f35d00b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:08.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.037+0000 7f35d721a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35d013ac20 0x7f35d00a94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:08.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.037+0000 7f35d721a700 10 monclient(hunting): picked mon.noname-b con 0x7f35d013ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:08.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.037+0000 7f35d721a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:08.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.037+0000 7f35d721a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00a88c0 con 0x7f35d00b1830 2023-09-30T00:49:08.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.037+0000 7f35d721a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:08.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.037+0000 7f35d721a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00ab480 con 0x7f35d013ac20 2023-09-30T00:49:08.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.037+0000 7f35d721a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:08.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.037+0000 7f35d721a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d00a8380 con 0x7f35d00b1460 2023-09-30T00:49:08.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.037+0000 7f35d721a700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:08.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.037+0000 7f35d721a700 10 monclient(hunting): authenticate will time out at 5409.338074s 2023-09-30T00:49:08.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.039+0000 7f35d4fb6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35d00b1460 0x7f35d013e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:08.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.040+0000 7f35d57b7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35d013ac20 0x7f35d00a94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:08.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.041+0000 7f35cffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35d00b1830 0x7f35d00a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:08.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.041+0000 7f35d4fb6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35d00b1460 0x7f35d013e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:47720/0 (socket says 172.21.15.40:47720) 2023-09-30T00:49:08.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.041+0000 7f35d4fb6700 1 -- 172.21.15.40:0/786190920 learned_addr learned my addr 172.21.15.40:0/786190920 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:08.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.041+0000 7f35d4fb6700 10 monclient(hunting): get_auth_request con 0x7f35d00b1460 auth_method 0 2023-09-30T00:49:08.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.041+0000 7f35d4fb6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:08.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.041+0000 7f35d4fb6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:08.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.041+0000 7f35d4fb6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:08.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.042+0000 7f35cffff700 10 monclient(hunting): get_auth_request con 0x7f35d00b1830 auth_method 0 2023-09-30T00:49:08.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.042+0000 7f35cffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:08.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.042+0000 7f35cffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:08.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.042+0000 7f35cffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:08.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.045+0000 7f35d4fb6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:08.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.045+0000 7f35d4fb6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:08.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.045+0000 7f35d4fb6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:08.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.046+0000 7f35cffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:08.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.046+0000 7f35cffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:08.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.046+0000 7f35cffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:08.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.049+0000 7f35d4fb6700 10 monclient(hunting): handle_auth_done global_id 4400 payload 824 2023-09-30T00:49:08.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.049+0000 7f35d4fb6700 1 -- 172.21.15.40:0/786190920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35d013ac20 msgr2=0x7f35d00a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:08.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.049+0000 7f35d4fb6700 1 --2- 172.21.15.40:0/786190920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35d013ac20 0x7f35d00a94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:08.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.049+0000 7f35d4fb6700 1 -- 172.21.15.40:0/786190920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35d00b1830 msgr2=0x7f35d00a7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:08.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.049+0000 7f35d4fb6700 1 --2- 172.21.15.40:0/786190920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35d00b1830 0x7f35d00a7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:08.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.049+0000 7f35d4fb6700 10 monclient: _finish_hunting 0 2023-09-30T00:49:08.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.049+0000 7f35d4fb6700 1 monclient: found mon.noname-c 2023-09-30T00:49:08.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.049+0000 7f35d4fb6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:08.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.050+0000 7f35d4fb6700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:49:08.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.050+0000 7f35d4fb6700 1 -- 172.21.15.40:0/786190920 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c8009a20 con 0x7f35d00b1460 2023-09-30T00:49:08.053 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:02:43.764 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:08.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.054+0000 7f35d4fb6700 1 --2- 172.21.15.40:0/786190920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35d00b1460 0x7f35d013e4f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f35c400c7f0 tx=0x7f35c4007f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:08.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.054+0000 7f35ccff9700 10 client.?.objecter ms_handle_connect 0x7f35d00b1460 2023-09-30T00:49:08.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.054+0000 7f35ccff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:08.060 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:02:43.772 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35ccff9700 1 -- 172.21.15.40:0/786190920 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4005b30 con 0x7f35d00b1460 2023-09-30T00:49:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35ccff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35ccff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35ccff9700 10 monclient: dump: 2023-09-30T00:49:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:08.082 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35ccff9700 10 monclient: _finish_auth 0 2023-09-30T00:49:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35ccff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35d721a700 5 monclient: authenticate success, global_id 4400 2023-09-30T00:49:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35ccff9700 1 -- 172.21.15.40:0/786190920 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c4005cb0 con 0x7f35d00b1460 2023-09-30T00:49:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35ccff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35ccff9700 1 -- 172.21.15.40:0/786190920 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c4004a80 con 0x7f35d00b1460 2023-09-30T00:49:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35d721a700 10 monclient: _renew_subs 2023-09-30T00:49:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35d721a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35d721a700 1 -- 172.21.15.40:0/786190920 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d00a8600 con 0x7f35d00b1460 2023-09-30T00:49:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35ccff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35ccff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35ccff9700 10 monclient: dump: 2023-09-30T00:49:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:08.084 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.079+0000 7f35ceffd700 4 set_mon_vals no callback set 2023-09-30T00:49:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.080+0000 7f35d721a700 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.080+0000 7f35d721a700 10 monclient: _renew_subs 2023-09-30T00:49:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.080+0000 7f35d721a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.080+0000 7f35d721a700 1 -- 172.21.15.40:0/786190920 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d00a9c60 con 0x7f35d00b1460 2023-09-30T00:49:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.080+0000 7f35d721a700 1 librados: init done 2023-09-30T00:49:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.080+0000 7f35d721a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.081+0000 7f35d721a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.081+0000 7f35d721a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.081+0000 7f35d721a700 1 -- 172.21.15.40:0/786190920 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d0064a00 con 0x7f35d00b1460 2023-09-30T00:49:08.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.387+0000 7f35ccff9700 1 -- 172.21.15.40:0/786190920 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f35c40070d0 con 0x7f35d00b1460 2023-09-30T00:49:08.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.387+0000 7f35ccff9700 1 --2- 172.21.15.40:0/786190920 >> v2:172.21.15.135:6800/116426 conn(0x7f35b4068960 0x7f35b406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:08.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.387+0000 7f35ccff9700 1 -- 172.21.15.40:0/786190920 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f35c4083e20 con 0x7f35d00b1460 2023-09-30T00:49:08.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.387+0000 7f35ccff9700 10 client.4400.objecter ms_dispatch 0x7f35d0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:08.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.387+0000 7f35ccff9700 3 client.4400.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:08.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.387+0000 7f35ccff9700 3 client.4400.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:08.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.387+0000 7f35ccff9700 20 client.4400.objecter dump_active .. 0 homeless 2023-09-30T00:49:08.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.387+0000 7f35cffff700 1 --2- 172.21.15.40:0/786190920 >> v2:172.21.15.135:6800/116426 conn(0x7f35b4068960 0x7f35b406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:08.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.387+0000 7f35cffff700 10 monclient: get_auth_request con 0x7f35b4068960 auth_method 0 2023-09-30T00:49:08.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.388+0000 7f35ccff9700 10 client.4400.objecter ms_handle_connect 0x7f35b4068960 2023-09-30T00:49:08.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.388+0000 7f35cffff700 1 --2- 172.21.15.40:0/786190920 >> v2:172.21.15.135:6800/116426 conn(0x7f35b4068960 0x7f35b406ae10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009bf0 tx=0x7f35bc0091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:08.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.408+0000 7f35ccff9700 1 -- 172.21.15.40:0/786190920 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f35c4051740 con 0x7f35d00b1460 2023-09-30T00:49:08.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.408+0000 7f35ccff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:08.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.408+0000 7f35ccff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:08.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.701+0000 7f35d721a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:08.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.701+0000 7f35d721a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:08.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.701+0000 7f35d721a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:08.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.701+0000 7f35d721a700 1 -- 172.21.15.40:0/786190920 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f35d00b3170 con 0x7f35d00b1460 2023-09-30T00:49:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.707+0000 7f35ccff9700 1 -- 172.21.15.40:0/786190920 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f35c4054d60 con 0x7f35d00b1460 2023-09-30T00:49:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.707+0000 7f35ccff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:08.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.707+0000 7f35ccff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:08.707 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:08.708 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 10 librados: watch_flush enter 2023-09-30T00:49:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 10 librados: watch_flush exit 2023-09-30T00:49:08.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 20 client.4400.objecter shutdown clearing up homeless session... 2023-09-30T00:49:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 10 client.4400.objecter successfully canceled tick 2023-09-30T00:49:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 1 -- 172.21.15.40:0/786190920 >> v2:172.21.15.135:6800/116426 conn(0x7f35b4068960 msgr2=0x7f35b406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 1 --2- 172.21.15.40:0/786190920 >> v2:172.21.15.135:6800/116426 conn(0x7f35b4068960 0x7f35b406ae10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009bf0 tx=0x7f35bc0091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 10 monclient: shutdown 2023-09-30T00:49:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 1 -- 172.21.15.40:0/786190920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35d00b1460 msgr2=0x7f35d013e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 1 --2- 172.21.15.40:0/786190920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35d00b1460 0x7f35d013e4f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f35c400c7f0 tx=0x7f35c4007f70 comp rx=0 tx=0).stop 2023-09-30T00:49:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 1 -- 172.21.15.40:0/786190920 shutdown_connections 2023-09-30T00:49:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 1 --2- 172.21.15.40:0/786190920 >> v2:172.21.15.135:6800/116426 conn(0x7f35b4068960 0x7f35b406ae10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 1 --2- 172.21.15.40:0/786190920 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f35d013ac20 0x7f35d00a94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:08.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 1 --2- 172.21.15.40:0/786190920 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f35d00b1460 0x7f35d013e4f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 1 --2- 172.21.15.40:0/786190920 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f35d00b1830 0x7f35d00a7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 1 -- 172.21.15.40:0/786190920 >> 172.21.15.40:0/786190920 conn(0x7f35d00631f0 msgr2=0x7f35d00afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 1 -- 172.21.15.40:0/786190920 shutdown_connections 2023-09-30T00:49:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 1 -- 172.21.15.40:0/786190920 wait complete. 2023-09-30T00:49:08.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:08.709+0000 7f35d721a700 1 librados: shutdown 2023-09-30T00:49:11.725 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:11.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.864+0000 7fb5f361e700 10 monclient: get_monmap_and_config 2023-09-30T00:49:11.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.864+0000 7fb5f361e700 10 monclient: build_initial_monmap 2023-09-30T00:49:11.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.864+0000 7fb5f361e700 10 monclient: monmap: 2023-09-30T00:49:11.865 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:11.865 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:11.865 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:11.865577+0000 2023-09-30T00:49:11.865 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:11.865577+0000 2023-09-30T00:49:11.866 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:11.866 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:11.866 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:11.866 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:11.866 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:11.866 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:11.866 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:11.866 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.865+0000 7fb5f361e700 1 Processor -- start 2023-09-30T00:49:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.865+0000 7fb5f361e700 1 -- start start 2023-09-30T00:49:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.865+0000 7fb5f361e700 10 monclient: init 2023-09-30T00:49:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.865+0000 7fb5f361e700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.865+0000 7fb5f361e700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-09-30T00:49:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.865+0000 7fb5f361e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5ec0b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:49:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.865+0000 7fb5f361e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5ec0b1830 addr v1:172.21.15.135:6789/0 2023-09-30T00:49:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.866+0000 7fb5f361e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5ec0a5fb0 0x7fb5ec13b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.866+0000 7fb5f361e700 10 monclient(hunting): picked mon.noname-a con 0x7fb5ec0a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:49:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.866+0000 7fb5f361e700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ec0a5b20 con 0x7fb5ec0b1460 2023-09-30T00:49:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.866+0000 7fb5f361e700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ec0a71f0 con 0x7fb5ec0b1830 2023-09-30T00:49:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.866+0000 7fb5f361e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.866+0000 7fb5f361e700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5ec141e40 con 0x7fb5ec0a5fb0 2023-09-30T00:49:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.866+0000 7fb5f361e700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.866+0000 7fb5f361e700 10 monclient(hunting): authenticate will time out at 5413.167048s 2023-09-30T00:49:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.866+0000 7fb5f1bbb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5ec0a5fb0 0x7fb5ec13b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.866+0000 7fb5f0bb9700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fb5ec0b1830 0x7fb5ec13ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:60506/0 (socket says 172.21.15.40:60506) 2023-09-30T00:49:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.867+0000 7fb5f0bb9700 1 -- 172.21.15.40:0/2617785093 learned_addr learned my addr 172.21.15.40:0/2617785093 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:49:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.867+0000 7fb5f1bbb700 10 monclient(hunting): get_auth_request con 0x7fb5ec0a5fb0 auth_method 0 2023-09-30T00:49:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.867+0000 7fb5f1bbb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.867+0000 7fb5f1bbb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.867+0000 7fb5f1bbb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.867+0000 7fb5f1bbb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.867+0000 7fb5f1bbb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.868+0000 7fb5f1bbb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.870+0000 7fb5e3fff700 1 -- 172.21.15.40:0/2617785093 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131551276 0 0) 0x7fb5ec0a71f0 con 0x7fb5ec0b1830 2023-09-30T00:49:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.870+0000 7fb5e3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.870+0000 7fb5e3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:11.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.870+0000 7fb5e3fff700 10 monclient(hunting): my global_id is 4501 2023-09-30T00:49:11.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.870+0000 7fb5e3fff700 1 -- 172.21.15.40:0/2617785093 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5d8003570 con 0x7fb5ec0b1830 2023-09-30T00:49:11.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.870+0000 7fb5f1bbb700 10 monclient(hunting): handle_auth_done global_id 4461 payload 274 2023-09-30T00:49:11.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.870+0000 7fb5f1bbb700 1 -- 172.21.15.40:0/2617785093 >> v1:172.21.15.135:6789/0 conn(0x7fb5ec0b1830 legacy=0x7fb5ec13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:11.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.870+0000 7fb5f1bbb700 1 -- 172.21.15.40:0/2617785093 >> v1:172.21.15.40:6789/0 conn(0x7fb5ec0b1460 legacy=0x7fb5ec141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:11.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.870+0000 7fb5f1bbb700 10 monclient: _finish_hunting 0 2023-09-30T00:49:11.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.870+0000 7fb5f1bbb700 1 monclient: found mon.noname-a 2023-09-30T00:49:11.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.870+0000 7fb5f1bbb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:11.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.870+0000 7fb5f1bbb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:49:11.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.870+0000 7fb5f1bbb700 1 -- 172.21.15.40:0/2617785093 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ec0a74d0 con 0x7fb5ec0a5fb0 2023-09-30T00:49:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.872+0000 7fb5f1bbb700 1 --2- 172.21.15.40:0/2617785093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ec0a5fb0 0x7fb5ec13b330 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009e90 tx=0x7fb5e4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a81b21b57a1a5897 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5e3fff700 1 -- 172.21.15.40:0/2617785093 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e400e070 con 0x7fb5ec0a5fb0 2023-09-30T00:49:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5e3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5e3fff700 10 monclient: dump: 2023-09-30T00:49:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:11.880 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5e3fff700 10 monclient: _finish_auth 0 2023-09-30T00:49:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5e3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5e3fff700 1 -- 172.21.15.40:0/2617785093 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e400bbb0 con 0x7fb5ec0a5fb0 2023-09-30T00:49:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5e3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5e3fff700 1 -- 172.21.15.40:0/2617785093 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4012430 con 0x7fb5ec0a5fb0 2023-09-30T00:49:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5f361e700 5 monclient: authenticate success, global_id 4461 2023-09-30T00:49:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5f361e700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5f361e700 4 set_mon_vals no callback set 2023-09-30T00:49:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5e3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.878+0000 7fb5e3fff700 10 monclient: dump: 2023-09-30T00:49:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:11.882 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.879+0000 7fb5f361e700 10 monclient: shutdown 2023-09-30T00:49:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.879+0000 7fb5f361e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.879+0000 7fb5f361e700 1 -- 172.21.15.40:0/2617785093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ec0a5fb0 msgr2=0x7fb5ec13b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.879+0000 7fb5f361e700 1 --2- 172.21.15.40:0/2617785093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ec0a5fb0 0x7fb5ec13b330 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009e90 tx=0x7fb5e4009780 comp rx=0 tx=0).stop 2023-09-30T00:49:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.879+0000 7fb5f361e700 1 -- 172.21.15.40:0/2617785093 shutdown_connections 2023-09-30T00:49:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.879+0000 7fb5f361e700 1 --2- 172.21.15.40:0/2617785093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ec0a5fb0 0x7fb5ec13b330 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.879+0000 7fb5f361e700 1 -- 172.21.15.40:0/2617785093 >> 172.21.15.40:0/2617785093 conn(0x7fb5ec0631f0 msgr2=0x7fb5ec0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.879+0000 7fb5f361e700 1 -- 172.21.15.40:0/2617785093 shutdown_connections 2023-09-30T00:49:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.879+0000 7fb5f361e700 1 -- 172.21.15.40:0/2617785093 wait complete. 2023-09-30T00:49:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.880+0000 7fb5f361e700 10 monclient: build_initial_monmap 2023-09-30T00:49:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.880+0000 7fb5f361e700 10 monclient: monmap: 2023-09-30T00:49:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:11.888 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.880+0000 7fb5f361e700 1 librados: starting msgr at 2023-09-30T00:49:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.880+0000 7fb5f361e700 1 librados: starting objecter 2023-09-30T00:49:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.881+0000 7fb5f361e700 1 Processor -- start 2023-09-30T00:49:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.881+0000 7fb5f361e700 1 -- start start 2023-09-30T00:49:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.881+0000 7fb5f361e700 1 librados: setting wanted keys 2023-09-30T00:49:11.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.881+0000 7fb5f361e700 1 librados: calling monclient init 2023-09-30T00:49:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.881+0000 7fb5f361e700 10 monclient: init 2023-09-30T00:49:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:49:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ec0a5fb0 0x7fb5ec13e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 10 monclient(hunting): picked mon.noname-a con 0x7fb5ec0a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5ec0b1460 0x7fb5ec0a79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 10 monclient(hunting): picked mon.noname-c con 0x7fb5ec0b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5ec0b1830 0x7fb5ec0a9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 10 monclient(hunting): picked mon.noname-b con 0x7fb5ec0b1830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec0a9e70 con 0x7fb5ec0a5fb0 2023-09-30T00:49:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec0a9bc0 con 0x7fb5ec0b1830 2023-09-30T00:49:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec0aba00 con 0x7fb5ec0b1460 2023-09-30T00:49:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f361e700 10 monclient(hunting): authenticate will time out at 5413.183426s 2023-09-30T00:49:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.882+0000 7fb5f13ba700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ec0a5fb0 0x7fb5ec13e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f0bb9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5ec0b1460 0x7fb5ec0a79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f1bbb700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5ec0b1830 0x7fb5ec0a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f13ba700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ec0a5fb0 0x7fb5ec13e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47382/0 (socket says 172.21.15.40:47382) 2023-09-30T00:49:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f13ba700 1 -- 172.21.15.40:0/2838890669 learned_addr learned my addr 172.21.15.40:0/2838890669 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f13ba700 10 monclient(hunting): get_auth_request con 0x7fb5ec0a5fb0 auth_method 0 2023-09-30T00:49:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f13ba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f13ba700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f13ba700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f1bbb700 10 monclient(hunting): get_auth_request con 0x7fb5ec0b1830 auth_method 0 2023-09-30T00:49:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f1bbb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f1bbb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f1bbb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f0bb9700 10 monclient(hunting): get_auth_request con 0x7fb5ec0b1460 auth_method 0 2023-09-30T00:49:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f0bb9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f0bb9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.883+0000 7fb5f0bb9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.884+0000 7fb5f13ba700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.884+0000 7fb5f13ba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.884+0000 7fb5f13ba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.884+0000 7fb5f1bbb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:11.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.884+0000 7fb5f1bbb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:11.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.884+0000 7fb5f1bbb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:11.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.884+0000 7fb5f0bb9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:11.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.884+0000 7fb5f0bb9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:11.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.884+0000 7fb5f0bb9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:11.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.888+0000 7fb5f13ba700 10 monclient(hunting): handle_auth_done global_id 4464 payload 824 2023-09-30T00:49:11.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.888+0000 7fb5f13ba700 1 -- 172.21.15.40:0/2838890669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5ec0b1460 msgr2=0x7fb5ec0a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:11.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.888+0000 7fb5f13ba700 1 --2- 172.21.15.40:0/2838890669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5ec0b1460 0x7fb5ec0a79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:11.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.888+0000 7fb5f13ba700 1 -- 172.21.15.40:0/2838890669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5ec0b1830 msgr2=0x7fb5ec0a9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:11.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.888+0000 7fb5f13ba700 1 --2- 172.21.15.40:0/2838890669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5ec0b1830 0x7fb5ec0a9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:11.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.888+0000 7fb5f13ba700 10 monclient: _finish_hunting 0 2023-09-30T00:49:11.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.888+0000 7fb5f13ba700 1 monclient: found mon.noname-a 2023-09-30T00:49:11.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.888+0000 7fb5f13ba700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:11.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.888+0000 7fb5f13ba700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:49:11.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.888+0000 7fb5f13ba700 1 -- 172.21.15.40:0/2838890669 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e4009a20 con 0x7fb5ec0a5fb0 2023-09-30T00:49:11.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.890+0000 7fb5f13ba700 1 --2- 172.21.15.40:0/2838890669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ec0a5fb0 0x7fb5ec13e660 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800b600 tx=0x7fb5e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:11.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.890+0000 7fb5e17fa700 10 client.?.objecter ms_handle_connect 0x7fb5ec0a5fb0 2023-09-30T00:49:11.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.890+0000 7fb5e17fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5e17fa700 1 -- 172.21.15.40:0/2838890669 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e800f510 con 0x7fb5ec0a5fb0 2023-09-30T00:49:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5e17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5e17fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5e17fa700 10 monclient: dump: 2023-09-30T00:49:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:11.899 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5e17fa700 10 monclient: _finish_auth 0 2023-09-30T00:49:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5e17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5e17fa700 1 -- 172.21.15.40:0/2838890669 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e800f690 con 0x7fb5ec0a5fb0 2023-09-30T00:49:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5f361e700 5 monclient: authenticate success, global_id 4464 2023-09-30T00:49:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5f361e700 10 monclient: _renew_subs 2023-09-30T00:49:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5f361e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:49:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5f361e700 1 -- 172.21.15.40:0/2838890669 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ec0abce0 con 0x7fb5ec0a5fb0 2023-09-30T00:49:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5e17fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5e17fa700 1 -- 172.21.15.40:0/2838890669 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e800fa50 con 0x7fb5ec0a5fb0 2023-09-30T00:49:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5e17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5e17fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5e17fa700 10 monclient: dump: 2023-09-30T00:49:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:11.901 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:11.901 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:11.901 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:11.901 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:11.901 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:11.901 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:11.901 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:11.901 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:11.901 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:11.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5e37fe700 4 set_mon_vals no callback set 2023-09-30T00:49:11.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5f361e700 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:11.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5f361e700 10 monclient: _renew_subs 2023-09-30T00:49:11.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5f361e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:49:11.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5f361e700 1 -- 172.21.15.40:0/2838890669 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ec0aa5f0 con 0x7fb5ec0a5fb0 2023-09-30T00:49:11.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.895+0000 7fb5f361e700 1 librados: init done 2023-09-30T00:49:11.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.896+0000 7fb5f361e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:11.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.896+0000 7fb5f361e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:11.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.896+0000 7fb5f361e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:49:11.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:11.896+0000 7fb5f361e700 1 -- 172.21.15.40:0/2838890669 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ec0aa0d0 con 0x7fb5ec0a5fb0 2023-09-30T00:49:12.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.212+0000 7fb5e17fa700 1 -- 172.21.15.40:0/2838890669 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fb5e8024430 con 0x7fb5ec0a5fb0 2023-09-30T00:49:12.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.212+0000 7fb5e17fa700 1 --2- 172.21.15.40:0/2838890669 >> v2:172.21.15.135:6800/116426 conn(0x7fb5d00689e0 0x7fb5d006ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:12.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.212+0000 7fb5e17fa700 1 -- 172.21.15.40:0/2838890669 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fb5e8084140 con 0x7fb5ec0a5fb0 2023-09-30T00:49:12.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.212+0000 7fb5e17fa700 10 client.4464.objecter ms_dispatch 0x7fb5ec001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:12.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.212+0000 7fb5e17fa700 3 client.4464.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.212+0000 7fb5e17fa700 3 client.4464.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.212+0000 7fb5e17fa700 20 client.4464.objecter dump_active .. 0 homeless 2023-09-30T00:49:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.212+0000 7fb5e17fa700 1 -- 172.21.15.40:0/2838890669 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb5e80845d0 con 0x7fb5ec0a5fb0 2023-09-30T00:49:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.212+0000 7fb5e17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.212+0000 7fb5e17fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.212+0000 7fb5f0bb9700 1 --2- 172.21.15.40:0/2838890669 >> v2:172.21.15.135:6800/116426 conn(0x7fb5d00689e0 0x7fb5d006ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.212+0000 7fb5f0bb9700 10 monclient: get_auth_request con 0x7fb5d00689e0 auth_method 0 2023-09-30T00:49:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.213+0000 7fb5f0bb9700 1 --2- 172.21.15.40:0/2838890669 >> v2:172.21.15.135:6800/116426 conn(0x7fb5d00689e0 0x7fb5d006ae90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc007fd0 tx=0x7fb5dc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.213+0000 7fb5e17fa700 10 client.4464.objecter ms_handle_connect 0x7fb5d00689e0 2023-09-30T00:49:12.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.485+0000 7fb5f361e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:12.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.485+0000 7fb5f361e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:12.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.485+0000 7fb5f361e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:49:12.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.485+0000 7fb5f361e700 1 -- 172.21.15.40:0/2838890669 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb5ec0aa0d0 con 0x7fb5ec0a5fb0 2023-09-30T00:49:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.491+0000 7fb5e17fa700 1 -- 172.21.15.40:0/2838890669 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb5ec0aa0d0 con 0x7fb5ec0a5fb0 2023-09-30T00:49:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.491+0000 7fb5e17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.491+0000 7fb5e17fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:12.493 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:12.493 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:12.495 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.493+0000 7fb5f361e700 10 librados: watch_flush enter 2023-09-30T00:49:12.495 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.493+0000 7fb5f361e700 10 librados: watch_flush exit 2023-09-30T00:49:12.495 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.493+0000 7fb5f361e700 20 client.4464.objecter shutdown clearing up homeless session... 2023-09-30T00:49:12.495 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.493+0000 7fb5f361e700 10 client.4464.objecter successfully canceled tick 2023-09-30T00:49:12.495 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.493+0000 7fb5f361e700 1 -- 172.21.15.40:0/2838890669 >> v2:172.21.15.135:6800/116426 conn(0x7fb5d00689e0 msgr2=0x7fb5d006ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:12.495 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.494+0000 7fb5f361e700 1 --2- 172.21.15.40:0/2838890669 >> v2:172.21.15.135:6800/116426 conn(0x7fb5d00689e0 0x7fb5d006ae90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc007fd0 tx=0x7fb5dc009040 comp rx=0 tx=0).stop 2023-09-30T00:49:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.494+0000 7fb5f361e700 10 monclient: shutdown 2023-09-30T00:49:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.494+0000 7fb5f361e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.494+0000 7fb5f361e700 1 -- 172.21.15.40:0/2838890669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ec0a5fb0 msgr2=0x7fb5ec13e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.494+0000 7fb5f361e700 1 --2- 172.21.15.40:0/2838890669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ec0a5fb0 0x7fb5ec13e660 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800b600 tx=0x7fb5e80091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.494+0000 7fb5f361e700 1 -- 172.21.15.40:0/2838890669 shutdown_connections 2023-09-30T00:49:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.494+0000 7fb5f361e700 1 --2- 172.21.15.40:0/2838890669 >> v2:172.21.15.135:6800/116426 conn(0x7fb5d00689e0 0x7fb5d006ae90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.494+0000 7fb5f361e700 1 --2- 172.21.15.40:0/2838890669 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5ec0b1830 0x7fb5ec0a9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.494+0000 7fb5f361e700 1 --2- 172.21.15.40:0/2838890669 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5ec0a5fb0 0x7fb5ec13e660 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.494+0000 7fb5f361e700 1 --2- 172.21.15.40:0/2838890669 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5ec0b1460 0x7fb5ec0a79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.494+0000 7fb5f361e700 1 -- 172.21.15.40:0/2838890669 >> 172.21.15.40:0/2838890669 conn(0x7fb5ec0631f0 msgr2=0x7fb5ec13f730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.494+0000 7fb5f361e700 1 -- 172.21.15.40:0/2838890669 shutdown_connections 2023-09-30T00:49:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.494+0000 7fb5f361e700 1 -- 172.21.15.40:0/2838890669 wait complete. 2023-09-30T00:49:12.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:12.494+0000 7fb5f361e700 1 librados: shutdown 2023-09-30T00:49:15.510 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:15.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.649+0000 7fcf32228700 10 monclient: get_monmap_and_config 2023-09-30T00:49:15.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.649+0000 7fcf32228700 10 monclient: build_initial_monmap 2023-09-30T00:49:15.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.649+0000 7fcf32228700 10 monclient: monmap: 2023-09-30T00:49:15.650 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:15.650 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:15.650 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:15.650643+0000 2023-09-30T00:49:15.651 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:15.650643+0000 2023-09-30T00:49:15.651 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:15.651 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:15.651 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:15.651 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:15.651 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:15.651 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:15.651 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:15.651 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:15.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.650+0000 7fcf32228700 1 Processor -- start 2023-09-30T00:49:15.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.650+0000 7fcf32228700 1 -- start start 2023-09-30T00:49:15.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.650+0000 7fcf32228700 10 monclient: init 2023-09-30T00:49:15.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf32228700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:15.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf32228700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-09-30T00:49:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf32228700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcf2c0b1460 0x7fcf2c0b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf32228700 10 monclient(hunting): picked mon.noname-c con 0x7fcf2c0b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:49:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf32228700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf2c0a5fb0 0x7fcf2c141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf32228700 10 monclient(hunting): picked mon.noname-b con 0x7fcf2c0a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf32228700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcf2c0af410 addr v1:172.21.15.135:6789/0 2023-09-30T00:49:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf32228700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2c0a71f0 con 0x7fcf2c0af410 2023-09-30T00:49:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf32228700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf32228700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2c0a74d0 con 0x7fcf2c0b1460 2023-09-30T00:49:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf32228700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:15.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf32228700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0a63c0 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf32228700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:15.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf32228700 10 monclient(hunting): authenticate will time out at 5416.952152s 2023-09-30T00:49:15.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.651+0000 7fcf2affd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf2c0a5fb0 0x7fcf2c141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:15.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.652+0000 7fcf2b7fe700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcf2c0b1460 0x7fcf2c0b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:15.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.652+0000 7fcf2affd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf2c0a5fb0 0x7fcf2c141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:47298/0 (socket says 172.21.15.40:47298) 2023-09-30T00:49:15.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.652+0000 7fcf2affd700 1 -- 172.21.15.40:0/3983955673 learned_addr learned my addr 172.21.15.40:0/3983955673 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:15.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.652+0000 7fcf2affd700 10 monclient(hunting): get_auth_request con 0x7fcf2c0a5fb0 auth_method 0 2023-09-30T00:49:15.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.652+0000 7fcf2affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:15.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.652+0000 7fcf2affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:15.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.652+0000 7fcf2affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:15.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.652+0000 7fcf2b7fe700 10 monclient(hunting): get_auth_request con 0x7fcf2c0b1460 auth_method 0 2023-09-30T00:49:15.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.652+0000 7fcf2b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:15.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.652+0000 7fcf2b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:15.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.652+0000 7fcf2b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:15.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.653+0000 7fcf2affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:15.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.653+0000 7fcf2affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:15.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.653+0000 7fcf2affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.654+0000 7fcf2b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.654+0000 7fcf2b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.654+0000 7fcf2b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.655+0000 7fcf2affd700 10 monclient(hunting): handle_auth_done global_id 4406 payload 274 2023-09-30T00:49:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.655+0000 7fcf2affd700 1 -- 172.21.15.40:0/3983955673 >> v2:172.21.15.135:3300/0 conn(0x7fcf2c0b1460 msgr2=0x7fcf2c0b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.655+0000 7fcf2affd700 1 --2- 172.21.15.40:0/3983955673 >> v2:172.21.15.135:3300/0 conn(0x7fcf2c0b1460 0x7fcf2c0b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.655+0000 7fcf2affd700 1 -- 172.21.15.40:0/3983955673 >> v1:172.21.15.135:6789/0 conn(0x7fcf2c0af410 legacy=0x7fcf2c0af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.655+0000 7fcf2affd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:15.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.655+0000 7fcf2affd700 1 monclient: found mon.noname-b 2023-09-30T00:49:15.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.655+0000 7fcf2affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:15.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.655+0000 7fcf2affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:49:15.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.655+0000 7fcf2affd700 1 -- 172.21.15.40:0/3983955673 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c0a56d0 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.657+0000 7fcf2affd700 1 --2- 172.21.15.40:0/3983955673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf2c0a5fb0 0x7fcf2c141730 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcf200099b0 tx=0x7fcf20009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=652acecf263564da server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:15.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf2a7fc700 1 -- 172.21.15.40:0/3983955673 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20009ea0 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf2a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf2a7fc700 10 monclient: dump: 2023-09-30T00:49:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:15.664 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf2a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf2a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf2a7fc700 1 -- 172.21.15.40:0/3983955673 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2000b860 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf2a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf2a7fc700 1 -- 172.21.15.40:0/3983955673 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2000bc00 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf32228700 5 monclient: authenticate success, global_id 4406 2023-09-30T00:49:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf32228700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf32228700 4 set_mon_vals no callback set 2023-09-30T00:49:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf2a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf2a7fc700 10 monclient: dump: 2023-09-30T00:49:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:15.665 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:15.668 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf32228700 10 monclient: shutdown 2023-09-30T00:49:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf32228700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf32228700 1 -- 172.21.15.40:0/3983955673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf2c0a5fb0 msgr2=0x7fcf2c141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf32228700 1 --2- 172.21.15.40:0/3983955673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf2c0a5fb0 0x7fcf2c141730 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcf200099b0 tx=0x7fcf20009250 comp rx=0 tx=0).stop 2023-09-30T00:49:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf32228700 1 -- 172.21.15.40:0/3983955673 shutdown_connections 2023-09-30T00:49:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf32228700 1 --2- 172.21.15.40:0/3983955673 >> v2:172.21.15.135:3300/0 conn(0x7fcf2c0b1460 0x7fcf2c0b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf32228700 1 --2- 172.21.15.40:0/3983955673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf2c0a5fb0 0x7fcf2c141730 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.662+0000 7fcf32228700 1 -- 172.21.15.40:0/3983955673 >> 172.21.15.40:0/3983955673 conn(0x7fcf2c0631f0 msgr2=0x7fcf2c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.663+0000 7fcf32228700 1 -- 172.21.15.40:0/3983955673 shutdown_connections 2023-09-30T00:49:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.663+0000 7fcf32228700 1 -- 172.21.15.40:0/3983955673 wait complete. 2023-09-30T00:49:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.663+0000 7fcf32228700 10 monclient: build_initial_monmap 2023-09-30T00:49:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.663+0000 7fcf32228700 10 monclient: monmap: 2023-09-30T00:49:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:15.670 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.664+0000 7fcf32228700 1 librados: starting msgr at 2023-09-30T00:49:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.664+0000 7fcf32228700 1 librados: starting objecter 2023-09-30T00:49:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.664+0000 7fcf32228700 1 Processor -- start 2023-09-30T00:49:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.664+0000 7fcf32228700 1 -- start start 2023-09-30T00:49:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.664+0000 7fcf32228700 1 librados: setting wanted keys 2023-09-30T00:49:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.664+0000 7fcf32228700 1 librados: calling monclient init 2023-09-30T00:49:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.664+0000 7fcf32228700 10 monclient: init 2023-09-30T00:49:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:49:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf2c0a5fb0 0x7fcf2c13e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 10 monclient(hunting): picked mon.noname-b con 0x7fcf2c0a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2c0af410 0x7fcf2c151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 10 monclient(hunting): picked mon.noname-a con 0x7fcf2c0af410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf2c0b1460 0x7fcf2c0a99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 10 monclient(hunting): picked mon.noname-c con 0x7fcf2c0b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0aea40 con 0x7fcf2c0af410 2023-09-30T00:49:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0ab990 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c151bd0 con 0x7fcf2c0b1460 2023-09-30T00:49:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.665+0000 7fcf32228700 10 monclient(hunting): authenticate will time out at 5416.966637s 2023-09-30T00:49:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.666+0000 7fcf2affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2c0af410 0x7fcf2c151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.666+0000 7fcf2b7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf2c0a5fb0 0x7fcf2c13e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.666+0000 7fcf2b7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf2c0a5fb0 0x7fcf2c13e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:51208/0 (socket says 172.21.15.40:51208) 2023-09-30T00:49:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.666+0000 7fcf2b7fe700 1 -- 172.21.15.40:0/3952726207 learned_addr learned my addr 172.21.15.40:0/3952726207 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.666+0000 7fcf2b7fe700 10 monclient(hunting): get_auth_request con 0x7fcf2c0a5fb0 auth_method 0 2023-09-30T00:49:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.666+0000 7fcf2b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.666+0000 7fcf2b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.666+0000 7fcf2b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.666+0000 7fcf2affd700 10 monclient(hunting): get_auth_request con 0x7fcf2c0af410 auth_method 0 2023-09-30T00:49:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.666+0000 7fcf2affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.666+0000 7fcf2affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.666+0000 7fcf2affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.667+0000 7fcf2b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.667+0000 7fcf2b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.667+0000 7fcf2b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.667+0000 7fcf2affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.667+0000 7fcf2affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.667+0000 7fcf2affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.668+0000 7fcf2bfff700 1 --2- 172.21.15.40:0/3952726207 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf2c0b1460 0x7fcf2c0a99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.668+0000 7fcf2bfff700 10 monclient(hunting): get_auth_request con 0x7fcf2c0b1460 auth_method 0 2023-09-30T00:49:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.668+0000 7fcf2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.668+0000 7fcf2bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.668+0000 7fcf2bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:15.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.669+0000 7fcf2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:15.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.669+0000 7fcf2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:15.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.669+0000 7fcf2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:15.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.670+0000 7fcf2b7fe700 10 monclient(hunting): handle_auth_done global_id 4510 payload 824 2023-09-30T00:49:15.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.670+0000 7fcf2b7fe700 1 -- 172.21.15.40:0/3952726207 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf2c0b1460 msgr2=0x7fcf2c0a99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:15.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.670+0000 7fcf2b7fe700 1 --2- 172.21.15.40:0/3952726207 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf2c0b1460 0x7fcf2c0a99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:15.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.670+0000 7fcf2b7fe700 1 -- 172.21.15.40:0/3952726207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2c0af410 msgr2=0x7fcf2c151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:15.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.670+0000 7fcf2b7fe700 1 --2- 172.21.15.40:0/3952726207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2c0af410 0x7fcf2c151690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:15.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.670+0000 7fcf2b7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:49:15.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.670+0000 7fcf2b7fe700 1 monclient: found mon.noname-b 2023-09-30T00:49:15.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.670+0000 7fcf2b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:15.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.670+0000 7fcf2b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:49:15.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.670+0000 7fcf2b7fe700 1 -- 172.21.15.40:0/3952726207 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf200094f0 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.673+0000 7fcf2b7fe700 1 --2- 172.21.15.40:0/3952726207 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf2c0a5fb0 0x7fcf2c13e610 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00e960 tx=0x7fcf1c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:15.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.673+0000 7fcf137fe700 10 client.?.objecter ms_handle_connect 0x7fcf2c0a5fb0 2023-09-30T00:49:15.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.673+0000 7fcf137fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:15.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf137fe700 1 -- 172.21.15.40:0/3952726207 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c013680 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:15.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf137fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf137fe700 10 monclient: dump: 2023-09-30T00:49:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:15.680 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf137fe700 10 monclient: _finish_auth 0 2023-09-30T00:49:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf137fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf137fe700 1 -- 172.21.15.40:0/3952726207 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1c0137e0 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf137fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf137fe700 1 -- 172.21.15.40:0/3952726207 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1c013b80 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf137fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:49:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf137fe700 10 monclient: dump: 2023-09-30T00:49:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:15.682 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf29ffb700 4 set_mon_vals no callback set 2023-09-30T00:49:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf32228700 5 monclient: authenticate success, global_id 4510 2023-09-30T00:49:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf32228700 10 monclient: _renew_subs 2023-09-30T00:49:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf32228700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.678+0000 7fcf32228700 1 -- 172.21.15.40:0/3952726207 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2c151eb0 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.679+0000 7fcf32228700 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.679+0000 7fcf32228700 10 monclient: _renew_subs 2023-09-30T00:49:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.679+0000 7fcf32228700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.679+0000 7fcf32228700 1 -- 172.21.15.40:0/3952726207 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c0aa150 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.679+0000 7fcf32228700 1 librados: init done 2023-09-30T00:49:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.679+0000 7fcf32228700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.679+0000 7fcf32228700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.679+0000 7fcf32228700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:15.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.679+0000 7fcf32228700 1 -- 172.21.15.40:0/3952726207 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2c152040 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.936 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.935+0000 7fcf137fe700 1 -- 172.21.15.40:0/3952726207 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fcf1c0253d0 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.936 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.935+0000 7fcf137fe700 1 --2- 172.21.15.40:0/3952726207 >> v2:172.21.15.135:6800/116426 conn(0x7fcf18068960 0x7fcf1806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:15.936 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.935+0000 7fcf137fe700 1 -- 172.21.15.40:0/3952726207 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fcf1c084520 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.936 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.935+0000 7fcf137fe700 10 client.4510.objecter ms_dispatch 0x7fcf2c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:15.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.935+0000 7fcf137fe700 3 client.4510.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:15.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.935+0000 7fcf137fe700 3 client.4510.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:15.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.935+0000 7fcf137fe700 20 client.4510.objecter dump_active .. 0 homeless 2023-09-30T00:49:15.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.936+0000 7fcf2affd700 1 --2- 172.21.15.40:0/3952726207 >> v2:172.21.15.135:6800/116426 conn(0x7fcf18068960 0x7fcf1806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:15.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.936+0000 7fcf2affd700 10 monclient: get_auth_request con 0x7fcf18068960 auth_method 0 2023-09-30T00:49:15.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.936+0000 7fcf2affd700 1 --2- 172.21.15.40:0/3952726207 >> v2:172.21.15.135:6800/116426 conn(0x7fcf18068960 0x7fcf1806ae10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcf200099b0 tx=0x7fcf20017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:15.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.936+0000 7fcf137fe700 10 client.4510.objecter ms_handle_connect 0x7fcf18068960 2023-09-30T00:49:15.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.980+0000 7fcf137fe700 1 -- 172.21.15.40:0/3952726207 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcf1c051e40 con 0x7fcf2c0a5fb0 2023-09-30T00:49:15.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.980+0000 7fcf137fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:15.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:15.980+0000 7fcf137fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:16.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.288+0000 7fcf32228700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:16.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.288+0000 7fcf32228700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:16.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.288+0000 7fcf32228700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:16.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.288+0000 7fcf32228700 1 -- 172.21.15.40:0/3952726207 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcf2c152040 con 0x7fcf2c0a5fb0 2023-09-30T00:49:16.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.294+0000 7fcf137fe700 1 -- 172.21.15.40:0/3952726207 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcf1c017100 con 0x7fcf2c0a5fb0 2023-09-30T00:49:16.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.294+0000 7fcf137fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:16.294 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.294+0000 7fcf137fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:16.295 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:16.295 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:16.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 10 librados: watch_flush enter 2023-09-30T00:49:16.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 10 librados: watch_flush exit 2023-09-30T00:49:16.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 20 client.4510.objecter shutdown clearing up homeless session... 2023-09-30T00:49:16.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 10 client.4510.objecter successfully canceled tick 2023-09-30T00:49:16.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 1 -- 172.21.15.40:0/3952726207 >> v2:172.21.15.135:6800/116426 conn(0x7fcf18068960 msgr2=0x7fcf1806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 1 --2- 172.21.15.40:0/3952726207 >> v2:172.21.15.135:6800/116426 conn(0x7fcf18068960 0x7fcf1806ae10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fcf200099b0 tx=0x7fcf20017000 comp rx=0 tx=0).stop 2023-09-30T00:49:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 10 monclient: shutdown 2023-09-30T00:49:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 1 -- 172.21.15.40:0/3952726207 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf2c0a5fb0 msgr2=0x7fcf2c13e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 1 --2- 172.21.15.40:0/3952726207 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf2c0a5fb0 0x7fcf2c13e610 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00e960 tx=0x7fcf1c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 1 -- 172.21.15.40:0/3952726207 shutdown_connections 2023-09-30T00:49:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 1 --2- 172.21.15.40:0/3952726207 >> v2:172.21.15.135:6800/116426 conn(0x7fcf18068960 0x7fcf1806ae10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 1 --2- 172.21.15.40:0/3952726207 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf2c0a5fb0 0x7fcf2c13e610 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 1 --2- 172.21.15.40:0/3952726207 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf2c0b1460 0x7fcf2c0a99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 1 --2- 172.21.15.40:0/3952726207 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf2c0af410 0x7fcf2c151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 1 -- 172.21.15.40:0/3952726207 >> 172.21.15.40:0/3952726207 conn(0x7fcf2c0631f0 msgr2=0x7fcf2c0635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:16.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 1 -- 172.21.15.40:0/3952726207 shutdown_connections 2023-09-30T00:49:16.299 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 1 -- 172.21.15.40:0/3952726207 wait complete. 2023-09-30T00:49:16.299 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:16.296+0000 7fcf32228700 1 librados: shutdown 2023-09-30T00:49:19.312 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.452+0000 7f562a51f700 10 monclient: get_monmap_and_config 2023-09-30T00:49:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.452+0000 7f562a51f700 10 monclient: build_initial_monmap 2023-09-30T00:49:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.452+0000 7f562a51f700 10 monclient: monmap: 2023-09-30T00:49:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:19.453650+0000 2023-09-30T00:49:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:19.453650+0000 2023-09-30T00:49:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:19.454 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:19.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.453+0000 7f562a51f700 1 Processor -- start 2023-09-30T00:49:19.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.453+0000 7f562a51f700 1 -- start start 2023-09-30T00:49:19.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.453+0000 7f562a51f700 10 monclient: init 2023-09-30T00:49:19.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f562a51f700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:19.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f562a51f700 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-09-30T00:49:19.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f562a51f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56240b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:49:19.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f562a51f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56240b1830 0x7f56240a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:19.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f562a51f700 10 monclient(hunting): picked mon.noname-b con 0x7f56240b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:19.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f562a51f700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f562413ac20 0x7f562413b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:19.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f562a51f700 10 monclient(hunting): picked mon.noname-c con 0x7f562413ac20 addr v2:172.21.15.135:3300/0 2023-09-30T00:49:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f562a51f700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56240a5b20 con 0x7f56240b1460 2023-09-30T00:49:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f562a51f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f562a51f700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5624141e40 con 0x7f562413ac20 2023-09-30T00:49:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f562a51f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f562a51f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56240a64f0 con 0x7f56240b1830 2023-09-30T00:49:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f562a51f700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f562a51f700 10 monclient(hunting): authenticate will time out at 5420.755152s 2023-09-30T00:49:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f56237fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56240b1830 0x7f56240a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.454+0000 7f5628abc700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f562413ac20 0x7f562413b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.455+0000 7f56237fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56240b1830 0x7f56240a5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:47318/0 (socket says 172.21.15.40:47318) 2023-09-30T00:49:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.455+0000 7f56237fe700 1 -- 172.21.15.40:0/2622754254 learned_addr learned my addr 172.21.15.40:0/2622754254 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.455+0000 7f56237fe700 10 monclient(hunting): get_auth_request con 0x7f56240b1830 auth_method 0 2023-09-30T00:49:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.455+0000 7f56237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.455+0000 7f56237fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.455+0000 7f56237fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.455+0000 7f5628abc700 10 monclient(hunting): get_auth_request con 0x7f562413ac20 auth_method 0 2023-09-30T00:49:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.455+0000 7f5628abc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.455+0000 7f5628abc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.455+0000 7f5628abc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.456+0000 7f56237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.456+0000 7f56237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.456+0000 7f56237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.456+0000 7f5628abc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.456+0000 7f5628abc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.456+0000 7f5628abc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.458+0000 7f56237fe700 10 monclient(hunting): handle_auth_done global_id 4412 payload 274 2023-09-30T00:49:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.458+0000 7f56237fe700 1 -- 172.21.15.40:0/2622754254 >> v2:172.21.15.135:3300/0 conn(0x7f562413ac20 msgr2=0x7f562413b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.458+0000 7f56237fe700 1 --2- 172.21.15.40:0/2622754254 >> v2:172.21.15.135:3300/0 conn(0x7f562413ac20 0x7f562413b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.458+0000 7f56237fe700 1 -- 172.21.15.40:0/2622754254 >> v1:172.21.15.135:6789/0 conn(0x7f56240b1460 legacy=0x7f5624141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.458+0000 7f56237fe700 10 monclient: _finish_hunting 0 2023-09-30T00:49:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.458+0000 7f56237fe700 1 monclient: found mon.noname-b 2023-09-30T00:49:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.458+0000 7f56237fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.458+0000 7f56237fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:49:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.458+0000 7f56237fe700 1 -- 172.21.15.40:0/2622754254 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56240a71f0 con 0x7f56240b1830 2023-09-30T00:49:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.460+0000 7f56237fe700 1 --2- 172.21.15.40:0/2622754254 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56240b1830 0x7f56240a5fb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f56180099b0 tx=0x7f5618009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12584f9afe43a846 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f5622ffd700 1 -- 172.21.15.40:0/2622754254 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5618009ea0 con 0x7f56240b1830 2023-09-30T00:49:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f5622ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f5622ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f5622ffd700 10 monclient: dump: 2023-09-30T00:49:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:19.465 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:19.466 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:19.466 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:19.466 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:19.466 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:19.466 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:19.466 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:19.466 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:19.466 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:19.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f5622ffd700 10 monclient: _finish_auth 0 2023-09-30T00:49:19.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f5622ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:19.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f5622ffd700 1 -- 172.21.15.40:0/2622754254 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561800b820 con 0x7f56240b1830 2023-09-30T00:49:19.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f5622ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:19.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f5622ffd700 1 -- 172.21.15.40:0/2622754254 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561800bbc0 con 0x7f56240b1830 2023-09-30T00:49:19.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f562a51f700 5 monclient: authenticate success, global_id 4412 2023-09-30T00:49:19.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f562a51f700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:19.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f5622ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:19.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f562a51f700 4 set_mon_vals no callback set 2023-09-30T00:49:19.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f5622ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:19.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.464+0000 7f5622ffd700 10 monclient: dump: 2023-09-30T00:49:19.467 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:19.467 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:19.467 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:19.467 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:19.467 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:19.467 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:19.467 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:19.468 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:19.468 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:19.468 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:19.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.465+0000 7f562a51f700 10 monclient: shutdown 2023-09-30T00:49:19.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.465+0000 7f562a51f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:19.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.465+0000 7f562a51f700 1 -- 172.21.15.40:0/2622754254 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56240b1830 msgr2=0x7f56240a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:19.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.465+0000 7f562a51f700 1 --2- 172.21.15.40:0/2622754254 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56240b1830 0x7f56240a5fb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f56180099b0 tx=0x7f5618009250 comp rx=0 tx=0).stop 2023-09-30T00:49:19.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.465+0000 7f562a51f700 1 -- 172.21.15.40:0/2622754254 shutdown_connections 2023-09-30T00:49:19.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.465+0000 7f562a51f700 1 --2- 172.21.15.40:0/2622754254 >> v2:172.21.15.135:3300/0 conn(0x7f562413ac20 0x7f562413b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:19.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.465+0000 7f562a51f700 1 --2- 172.21.15.40:0/2622754254 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56240b1830 0x7f56240a5fb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:19.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.465+0000 7f562a51f700 1 -- 172.21.15.40:0/2622754254 >> 172.21.15.40:0/2622754254 conn(0x7f56240631f0 msgr2=0x7f56240635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:19.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.465+0000 7f562a51f700 1 -- 172.21.15.40:0/2622754254 shutdown_connections 2023-09-30T00:49:19.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.465+0000 7f562a51f700 1 -- 172.21.15.40:0/2622754254 wait complete. 2023-09-30T00:49:19.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.466+0000 7f562a51f700 10 monclient: build_initial_monmap 2023-09-30T00:49:19.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.466+0000 7f562a51f700 10 monclient: monmap: 2023-09-30T00:49:19.472 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:19.472 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:19.472 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:19.472 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:19.472 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:19.472 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:19.472 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:19.472 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:19.472 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:19.473 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:19.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.466+0000 7f562a51f700 1 librados: starting msgr at 2023-09-30T00:49:19.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.466+0000 7f562a51f700 1 librados: starting objecter 2023-09-30T00:49:19.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.467+0000 7f562a51f700 1 Processor -- start 2023-09-30T00:49:19.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.467+0000 7f562a51f700 1 -- start start 2023-09-30T00:49:19.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.467+0000 7f562a51f700 1 librados: setting wanted keys 2023-09-30T00:49:19.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.467+0000 7f562a51f700 1 librados: calling monclient init 2023-09-30T00:49:19.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.467+0000 7f562a51f700 10 monclient: init 2023-09-30T00:49:19.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:19.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:49:19.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56240b1460 0x7f562413dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:19.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 10 monclient(hunting): picked mon.noname-a con 0x7f56240b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:19.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56240b1830 0x7f56240a7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 10 monclient(hunting): picked mon.noname-c con 0x7f56240b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f562413ac20 0x7f56241398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 10 monclient(hunting): picked mon.noname-b con 0x7f562413ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56240ab240 con 0x7f56240b1460 2023-09-30T00:49:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562413e1e0 con 0x7f562413ac20 2023-09-30T00:49:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56240a8390 con 0x7f56240b1830 2023-09-30T00:49:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f562a51f700 10 monclient(hunting): authenticate will time out at 5420.769283s 2023-09-30T00:49:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f56237fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56240b1830 0x7f56240a7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f5628abc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f562413ac20 0x7f56241398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.468+0000 7f5623fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56240b1460 0x7f562413dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f5628abc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f562413ac20 0x7f56241398c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:51220/0 (socket says 172.21.15.40:51220) 2023-09-30T00:49:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f5628abc700 1 -- 172.21.15.40:0/1713657985 learned_addr learned my addr 172.21.15.40:0/1713657985 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f5628abc700 10 monclient(hunting): get_auth_request con 0x7f562413ac20 auth_method 0 2023-09-30T00:49:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f56237fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56240b1830 0x7f56240a7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:47332/0 (socket says 172.21.15.40:47332) 2023-09-30T00:49:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f5628abc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f5628abc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f5628abc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f56237fe700 10 monclient(hunting): get_auth_request con 0x7f56240b1830 auth_method 0 2023-09-30T00:49:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f56237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f56237fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f56237fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f5623fff700 10 monclient(hunting): get_auth_request con 0x7f56240b1460 auth_method 0 2023-09-30T00:49:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f5623fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f5623fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f5623fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f5628abc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f5628abc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f5628abc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f56237fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f56237fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.469+0000 7f56237fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.470+0000 7f5623fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.470+0000 7f5623fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.470+0000 7f5623fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:19.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.473+0000 7f5628abc700 10 monclient(hunting): handle_auth_done global_id 4516 payload 824 2023-09-30T00:49:19.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.473+0000 7f5628abc700 1 -- 172.21.15.40:0/1713657985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56240b1830 msgr2=0x7f56240a7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:19.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.473+0000 7f5628abc700 1 --2- 172.21.15.40:0/1713657985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56240b1830 0x7f56240a7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:19.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.473+0000 7f5628abc700 1 -- 172.21.15.40:0/1713657985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56240b1460 msgr2=0x7f562413dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:19.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.473+0000 7f5628abc700 1 --2- 172.21.15.40:0/1713657985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56240b1460 0x7f562413dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:19.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.473+0000 7f5628abc700 10 monclient: _finish_hunting 0 2023-09-30T00:49:19.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.473+0000 7f5628abc700 1 monclient: found mon.noname-b 2023-09-30T00:49:19.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.473+0000 7f5628abc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:19.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.473+0000 7f5628abc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:49:19.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.473+0000 7f5628abc700 1 -- 172.21.15.40:0/1713657985 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56180094f0 con 0x7f562413ac20 2023-09-30T00:49:19.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.475+0000 7f5628abc700 1 --2- 172.21.15.40:0/1713657985 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f562413ac20 0x7f56241398c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f561c009df0 tx=0x7f561c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:19.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.475+0000 7f560bfff700 10 client.?.objecter ms_handle_connect 0x7f562413ac20 2023-09-30T00:49:19.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.475+0000 7f560bfff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:19.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.479+0000 7f560bfff700 1 -- 172.21.15.40:0/1713657985 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c00caa0 con 0x7f562413ac20 2023-09-30T00:49:19.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.479+0000 7f560bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:19.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f560bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:19.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f560bfff700 10 monclient: dump: 2023-09-30T00:49:19.481 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:19.481 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:19.481 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:19.481 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:19.481 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:19.481 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:19.481 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:19.481 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:19.482 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f560bfff700 10 monclient: _finish_auth 0 2023-09-30T00:49:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f560bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f562a51f700 5 monclient: authenticate success, global_id 4516 2023-09-30T00:49:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f560bfff700 1 -- 172.21.15.40:0/1713657985 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561c00f040 con 0x7f562413ac20 2023-09-30T00:49:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f562a51f700 10 monclient: _renew_subs 2023-09-30T00:49:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f562a51f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f562a51f700 1 -- 172.21.15.40:0/1713657985 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56240a8550 con 0x7f562413ac20 2023-09-30T00:49:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f560bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f560bfff700 1 -- 172.21.15.40:0/1713657985 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c00ce70 con 0x7f562413ac20 2023-09-30T00:49:19.482 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f560bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f560bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:49:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f560bfff700 10 monclient: dump: 2023-09-30T00:49:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:19.483 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f562a51f700 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f562a51f700 10 monclient: _renew_subs 2023-09-30T00:49:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f562a51f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f562a51f700 1 -- 172.21.15.40:0/1713657985 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562413a040 con 0x7f562413ac20 2023-09-30T00:49:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f56227fc700 4 set_mon_vals no callback set 2023-09-30T00:49:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.480+0000 7f562a51f700 1 librados: init done 2023-09-30T00:49:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.481+0000 7f562a51f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:19.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.481+0000 7f562a51f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:19.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.481+0000 7f562a51f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:19.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.481+0000 7f562a51f700 1 -- 172.21.15.40:0/1713657985 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562413e3a0 con 0x7f562413ac20 2023-09-30T00:49:19.507 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.507+0000 7f560bfff700 1 -- 172.21.15.40:0/1713657985 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f561c014690 con 0x7f562413ac20 2023-09-30T00:49:19.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.507+0000 7f560bfff700 1 --2- 172.21.15.40:0/1713657985 >> v2:172.21.15.135:6800/116426 conn(0x7f5610068910 0x7f561006adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:19.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.507+0000 7f560bfff700 1 -- 172.21.15.40:0/1713657985 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f561c084ec0 con 0x7f562413ac20 2023-09-30T00:49:19.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.507+0000 7f560bfff700 10 client.4516.objecter ms_dispatch 0x7f5624001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:19.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.507+0000 7f560bfff700 3 client.4516.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:19.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.507+0000 7f560bfff700 3 client.4516.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:19.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.507+0000 7f560bfff700 20 client.4516.objecter dump_active .. 0 homeless 2023-09-30T00:49:19.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.507+0000 7f5623fff700 1 --2- 172.21.15.40:0/1713657985 >> v2:172.21.15.135:6800/116426 conn(0x7f5610068910 0x7f561006adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:19.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.507+0000 7f5623fff700 10 monclient: get_auth_request con 0x7f5610068910 auth_method 0 2023-09-30T00:49:19.509 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.508+0000 7f560bfff700 10 client.4516.objecter ms_handle_connect 0x7f5610068910 2023-09-30T00:49:19.509 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.508+0000 7f5623fff700 1 --2- 172.21.15.40:0/1713657985 >> v2:172.21.15.135:6800/116426 conn(0x7f5610068910 0x7f561006adc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5614003a30 tx=0x7f5614007ea0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:19.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.746+0000 7f560bfff700 1 -- 172.21.15.40:0/1713657985 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f561c052590 con 0x7f562413ac20 2023-09-30T00:49:19.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.746+0000 7f560bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:19.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:19.746+0000 7f560bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:20.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.007+0000 7f562a51f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:20.008 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.007+0000 7f562a51f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:20.008 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.007+0000 7f562a51f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:20.008 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.007+0000 7f562a51f700 1 -- 172.21.15.40:0/1713657985 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5624064a00 con 0x7f562413ac20 2023-09-30T00:49:20.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.013+0000 7f560bfff700 1 -- 172.21.15.40:0/1713657985 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f561c018100 con 0x7f562413ac20 2023-09-30T00:49:20.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.013+0000 7f560bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:20.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.013+0000 7f560bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:20.013 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:20.013 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:20.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.014+0000 7f562a51f700 10 librados: watch_flush enter 2023-09-30T00:49:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.014+0000 7f562a51f700 10 librados: watch_flush exit 2023-09-30T00:49:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.014+0000 7f562a51f700 20 client.4516.objecter shutdown clearing up homeless session... 2023-09-30T00:49:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.014+0000 7f562a51f700 10 client.4516.objecter successfully canceled tick 2023-09-30T00:49:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.014+0000 7f562a51f700 1 -- 172.21.15.40:0/1713657985 >> v2:172.21.15.135:6800/116426 conn(0x7f5610068910 msgr2=0x7f561006adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.014+0000 7f562a51f700 1 --2- 172.21.15.40:0/1713657985 >> v2:172.21.15.135:6800/116426 conn(0x7f5610068910 0x7f561006adc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5614003a30 tx=0x7f5614007ea0 comp rx=0 tx=0).stop 2023-09-30T00:49:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.014+0000 7f562a51f700 10 monclient: shutdown 2023-09-30T00:49:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.014+0000 7f562a51f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.014+0000 7f562a51f700 1 -- 172.21.15.40:0/1713657985 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f562413ac20 msgr2=0x7f56241398c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.014+0000 7f562a51f700 1 --2- 172.21.15.40:0/1713657985 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f562413ac20 0x7f56241398c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f561c009df0 tx=0x7f561c00b040 comp rx=0 tx=0).stop 2023-09-30T00:49:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.015+0000 7f562a51f700 1 -- 172.21.15.40:0/1713657985 shutdown_connections 2023-09-30T00:49:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.015+0000 7f562a51f700 1 --2- 172.21.15.40:0/1713657985 >> v2:172.21.15.135:6800/116426 conn(0x7f5610068910 0x7f561006adc0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:20.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.015+0000 7f562a51f700 1 --2- 172.21.15.40:0/1713657985 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f562413ac20 0x7f56241398c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:20.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.015+0000 7f562a51f700 1 --2- 172.21.15.40:0/1713657985 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56240b1460 0x7f562413dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:20.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.015+0000 7f562a51f700 1 --2- 172.21.15.40:0/1713657985 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56240b1830 0x7f56240a7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:20.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.015+0000 7f562a51f700 1 -- 172.21.15.40:0/1713657985 >> 172.21.15.40:0/1713657985 conn(0x7f56240631f0 msgr2=0x7f56240635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:20.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.015+0000 7f562a51f700 1 -- 172.21.15.40:0/1713657985 shutdown_connections 2023-09-30T00:49:20.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.015+0000 7f562a51f700 1 -- 172.21.15.40:0/1713657985 wait complete. 2023-09-30T00:49:20.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:20.015+0000 7f562a51f700 1 librados: shutdown 2023-09-30T00:49:23.029 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.162+0000 7f3941bee700 10 monclient: get_monmap_and_config 2023-09-30T00:49:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.162+0000 7f3941bee700 10 monclient: build_initial_monmap 2023-09-30T00:49:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.162+0000 7f3941bee700 10 monclient: monmap: 2023-09-30T00:49:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:23.163525+0000 2023-09-30T00:49:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:23.163525+0000 2023-09-30T00:49:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:23.164 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.163+0000 7f3941bee700 1 Processor -- start 2023-09-30T00:49:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.163+0000 7f3941bee700 1 -- start start 2023-09-30T00:49:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.163+0000 7f3941bee700 10 monclient: init 2023-09-30T00:49:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.163+0000 7f3941bee700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.163+0000 7f3941bee700 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-09-30T00:49:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.163+0000 7f3941bee700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f393c0b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:49:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.164+0000 7f3941bee700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f393c0b1830 0x7f393c0a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.164+0000 7f3941bee700 10 monclient(hunting): picked mon.noname-c con 0x7f393c0b1830 addr v2:172.21.15.135:3300/0 2023-09-30T00:49:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.164+0000 7f3941bee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f393c13ac20 0x7f393c13b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.164+0000 7f3941bee700 10 monclient(hunting): picked mon.noname-b con 0x7f393c13ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.164+0000 7f3941bee700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393c0a5b20 con 0x7f393c0b1460 2023-09-30T00:49:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.164+0000 7f3941bee700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.164+0000 7f3941bee700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393c141e40 con 0x7f393c0b1830 2023-09-30T00:49:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.164+0000 7f3941bee700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.164+0000 7f3941bee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0a64f0 con 0x7f393c13ac20 2023-09-30T00:49:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.164+0000 7f3941bee700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.164+0000 7f3941bee700 10 monclient(hunting): authenticate will time out at 5424.465093s 2023-09-30T00:49:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.164+0000 7f393affd700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f393c0b1830 0x7f393c0a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.164+0000 7f393bfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f393c13ac20 0x7f393c13b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393affd700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f393c0b1830 0x7f393c0a5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:50678/0 (socket says 172.21.15.40:50678) 2023-09-30T00:49:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393affd700 1 -- 172.21.15.40:0/2130693464 learned_addr learned my addr 172.21.15.40:0/2130693464 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393affd700 10 monclient(hunting): get_auth_request con 0x7f393c0b1830 auth_method 0 2023-09-30T00:49:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393bfff700 10 monclient(hunting): get_auth_request con 0x7f393c13ac20 auth_method 0 2023-09-30T00:49:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:23.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.165+0000 7f393bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:23.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.168+0000 7f393bfff700 10 monclient(hunting): handle_auth_done global_id 4418 payload 274 2023-09-30T00:49:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.168+0000 7f393bfff700 1 -- 172.21.15.40:0/2130693464 >> v2:172.21.15.135:3300/0 conn(0x7f393c0b1830 msgr2=0x7f393c0a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.168+0000 7f393bfff700 1 --2- 172.21.15.40:0/2130693464 >> v2:172.21.15.135:3300/0 conn(0x7f393c0b1830 0x7f393c0a5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.168+0000 7f393bfff700 1 -- 172.21.15.40:0/2130693464 >> v1:172.21.15.135:6789/0 conn(0x7f393c0b1460 legacy=0x7f393c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.168+0000 7f393bfff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.168+0000 7f393bfff700 1 monclient: found mon.noname-b 2023-09-30T00:49:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.168+0000 7f393bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.168+0000 7f393bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:49:23.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.168+0000 7f393bfff700 1 -- 172.21.15.40:0/2130693464 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c0a71f0 con 0x7f393c13ac20 2023-09-30T00:49:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.170+0000 7f393bfff700 1 --2- 172.21.15.40:0/2130693464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f393c13ac20 0x7f393c13b080 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3934009e60 tx=0x7f3934009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f378d09533cffb2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:23.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f393a7fc700 1 -- 172.21.15.40:0/2130693464 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393400e070 con 0x7f393c13ac20 2023-09-30T00:49:23.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f393a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:23.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f393a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:23.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f393a7fc700 10 monclient: dump: 2023-09-30T00:49:23.176 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:23.176 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:23.176 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:23.176 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:23.176 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:23.176 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:23.177 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:23.177 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:23.177 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:23.177 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:23.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f393a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:23.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f393a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:23.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f393a7fc700 1 -- 172.21.15.40:0/2130693464 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393400bb60 con 0x7f393c13ac20 2023-09-30T00:49:23.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f393a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:23.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f393a7fc700 1 -- 172.21.15.40:0/2130693464 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3934012430 con 0x7f393c13ac20 2023-09-30T00:49:23.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f3941bee700 5 monclient: authenticate success, global_id 4418 2023-09-30T00:49:23.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f3941bee700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:23.178 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f3941bee700 4 set_mon_vals no callback set 2023-09-30T00:49:23.178 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f393a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:23.178 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f393a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:23.178 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.174+0000 7f393a7fc700 10 monclient: dump: 2023-09-30T00:49:23.178 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:23.178 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:23.178 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:23.178 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:23.178 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:23.178 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:23.178 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:23.179 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:23.179 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:23.179 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:23.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.175+0000 7f3941bee700 10 monclient: shutdown 2023-09-30T00:49:23.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.175+0000 7f3941bee700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:23.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.175+0000 7f3941bee700 1 -- 172.21.15.40:0/2130693464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f393c13ac20 msgr2=0x7f393c13b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:23.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.175+0000 7f3941bee700 1 --2- 172.21.15.40:0/2130693464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f393c13ac20 0x7f393c13b080 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3934009e60 tx=0x7f3934009780 comp rx=0 tx=0).stop 2023-09-30T00:49:23.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.175+0000 7f3941bee700 1 -- 172.21.15.40:0/2130693464 shutdown_connections 2023-09-30T00:49:23.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.175+0000 7f3941bee700 1 --2- 172.21.15.40:0/2130693464 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f393c13ac20 0x7f393c13b080 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.175+0000 7f3941bee700 1 --2- 172.21.15.40:0/2130693464 >> v2:172.21.15.135:3300/0 conn(0x7f393c0b1830 0x7f393c0a5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.179 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.175+0000 7f3941bee700 1 -- 172.21.15.40:0/2130693464 >> 172.21.15.40:0/2130693464 conn(0x7f393c0631f0 msgr2=0x7f393c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:23.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.175+0000 7f3941bee700 1 -- 172.21.15.40:0/2130693464 shutdown_connections 2023-09-30T00:49:23.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.175+0000 7f3941bee700 1 -- 172.21.15.40:0/2130693464 wait complete. 2023-09-30T00:49:23.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.176+0000 7f3941bee700 10 monclient: build_initial_monmap 2023-09-30T00:49:23.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.176+0000 7f3941bee700 10 monclient: monmap: 2023-09-30T00:49:23.182 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:23.182 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:23.183 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:23.183 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:23.183 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:23.183 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:23.183 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:23.183 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:23.183 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:23.183 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:23.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.176+0000 7f3941bee700 1 librados: starting msgr at 2023-09-30T00:49:23.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.176+0000 7f3941bee700 1 librados: starting objecter 2023-09-30T00:49:23.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.177+0000 7f3941bee700 1 Processor -- start 2023-09-30T00:49:23.183 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.177+0000 7f3941bee700 1 -- start start 2023-09-30T00:49:23.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.177+0000 7f3941bee700 1 librados: setting wanted keys 2023-09-30T00:49:23.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.177+0000 7f3941bee700 1 librados: calling monclient init 2023-09-30T00:49:23.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.177+0000 7f3941bee700 10 monclient: init 2023-09-30T00:49:23.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:23.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:49:23.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f393c0b1460 0x7f393c13e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:23.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 10 monclient(hunting): picked mon.noname-b con 0x7f393c0b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:23.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f393c0b1830 0x7f393c150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:23.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 10 monclient(hunting): picked mon.noname-a con 0x7f393c0b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:23.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f393c13ac20 0x7f393c150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:23.184 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 10 monclient(hunting): picked mon.noname-c con 0x7f393c13ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:23.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:23.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0a9b70 con 0x7f393c0b1830 2023-09-30T00:49:23.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:23.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0ab270 con 0x7f393c0b1460 2023-09-30T00:49:23.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:23.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393c13e980 con 0x7f393c13ac20 2023-09-30T00:49:23.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:23.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f3941bee700 10 monclient(hunting): authenticate will time out at 5424.479130s 2023-09-30T00:49:23.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f393b7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f393c0b1460 0x7f393c13e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:23.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f393bfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f393c13ac20 0x7f393c150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:23.185 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.178+0000 7f393affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f393c0b1830 0x7f393c150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:23.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393b7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f393c0b1460 0x7f393c13e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:50682/0 (socket says 172.21.15.40:50682) 2023-09-30T00:49:23.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393b7fe700 1 -- 172.21.15.40:0/3883763286 learned_addr learned my addr 172.21.15.40:0/3883763286 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:23.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393b7fe700 10 monclient(hunting): get_auth_request con 0x7f393c0b1460 auth_method 0 2023-09-30T00:49:23.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:23.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:23.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:23.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393bfff700 10 monclient(hunting): get_auth_request con 0x7f393c13ac20 auth_method 0 2023-09-30T00:49:23.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:23.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:23.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:23.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393affd700 10 monclient(hunting): get_auth_request con 0x7f393c0b1830 auth_method 0 2023-09-30T00:49:23.186 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:23.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:23.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:23.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:23.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:23.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:23.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:23.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:23.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.179+0000 7f393bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:23.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.180+0000 7f393affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:23.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.180+0000 7f393affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:23.187 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.180+0000 7f393affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:23.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.182+0000 7f393b7fe700 10 monclient(hunting): handle_auth_done global_id 4522 payload 824 2023-09-30T00:49:23.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.182+0000 7f393b7fe700 1 -- 172.21.15.40:0/3883763286 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f393c13ac20 msgr2=0x7f393c150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:23.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.182+0000 7f393b7fe700 1 --2- 172.21.15.40:0/3883763286 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f393c13ac20 0x7f393c150e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.182+0000 7f393b7fe700 1 -- 172.21.15.40:0/3883763286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f393c0b1830 msgr2=0x7f393c150900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:23.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.182+0000 7f393b7fe700 1 --2- 172.21.15.40:0/3883763286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f393c0b1830 0x7f393c150900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.182+0000 7f393b7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:49:23.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.182+0000 7f393b7fe700 1 monclient: found mon.noname-b 2023-09-30T00:49:23.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.182+0000 7f393b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:23.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.182+0000 7f393b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:49:23.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.182+0000 7f393b7fe700 1 -- 172.21.15.40:0/3883763286 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3934009a20 con 0x7f393c0b1460 2023-09-30T00:49:23.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.185+0000 7f393b7fe700 1 --2- 172.21.15.40:0/3883763286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f393c0b1460 0x7f393c13e440 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f392400c590 tx=0x7f392400c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:23.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.185+0000 7f39337fe700 10 client.?.objecter ms_handle_connect 0x7f393c0b1460 2023-09-30T00:49:23.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.185+0000 7f39337fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:23.191 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f39337fe700 1 -- 172.21.15.40:0/3883763286 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924011590 con 0x7f393c0b1460 2023-09-30T00:49:23.191 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f39337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:23.191 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f39337fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:23.191 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f39337fe700 10 monclient: dump: 2023-09-30T00:49:23.191 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:23.192 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:23.192 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:23.192 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:23.192 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:23.192 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:23.192 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:23.192 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:23.192 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:23.192 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:23.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f39337fe700 10 monclient: _finish_auth 0 2023-09-30T00:49:23.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f39337fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f39337fe700 1 -- 172.21.15.40:0/3883763286 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3924011710 con 0x7f393c0b1460 2023-09-30T00:49:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f3941bee700 5 monclient: authenticate success, global_id 4522 2023-09-30T00:49:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f3941bee700 10 monclient: _renew_subs 2023-09-30T00:49:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f3941bee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f3941bee700 1 -- 172.21.15.40:0/3883763286 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393c0a9770 con 0x7f393c0b1460 2023-09-30T00:49:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f39337fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f39337fe700 1 -- 172.21.15.40:0/3883763286 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924011ad0 con 0x7f393c0b1460 2023-09-30T00:49:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f39337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f39337fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:49:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f39337fe700 10 monclient: dump: 2023-09-30T00:49:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:23.194 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f3939ffb700 4 set_mon_vals no callback set 2023-09-30T00:49:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f3941bee700 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:23.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f3941bee700 10 monclient: _renew_subs 2023-09-30T00:49:23.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f3941bee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:23.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f3941bee700 1 -- 172.21.15.40:0/3883763286 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393c0a8c10 con 0x7f393c0b1460 2023-09-30T00:49:23.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.189+0000 7f3941bee700 1 librados: init done 2023-09-30T00:49:23.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.190+0000 7f3941bee700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:23.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.190+0000 7f3941bee700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:23.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.190+0000 7f3941bee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:23.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.190+0000 7f3941bee700 1 -- 172.21.15.40:0/3883763286 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f393c0a9960 con 0x7f393c0b1460 2023-09-30T00:49:23.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.209+0000 7f39337fe700 1 -- 172.21.15.40:0/3883763286 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f392401b3b0 con 0x7f393c0b1460 2023-09-30T00:49:23.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.209+0000 7f39337fe700 1 --2- 172.21.15.40:0/3883763286 >> v2:172.21.15.135:6800/116426 conn(0x7f391c068890 0x7f391c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:23.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.210+0000 7f393affd700 1 --2- 172.21.15.40:0/3883763286 >> v2:172.21.15.135:6800/116426 conn(0x7f391c068890 0x7f391c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:23.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.210+0000 7f393affd700 10 monclient: get_auth_request con 0x7f391c068890 auth_method 0 2023-09-30T00:49:23.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.210+0000 7f39337fe700 10 client.4522.objecter ms_handle_connect 0x7f391c068890 2023-09-30T00:49:23.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.210+0000 7f393affd700 1 --2- 172.21.15.40:0/3883763286 >> v2:172.21.15.135:6800/116426 conn(0x7f391c068890 0x7f391c06ad40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f393c0b3fa0 tx=0x7f392c009250 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:23.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.214+0000 7f39337fe700 1 -- 172.21.15.40:0/3883763286 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3924055260 con 0x7f393c0b1460 2023-09-30T00:49:23.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.214+0000 7f39337fe700 10 client.4522.objecter ms_dispatch 0x7f393c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:23.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.214+0000 7f39337fe700 3 client.4522.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:23.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.214+0000 7f39337fe700 3 client.4522.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:23.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.214+0000 7f39337fe700 20 client.4522.objecter dump_active .. 0 homeless 2023-09-30T00:49:23.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.451+0000 7f39337fe700 1 -- 172.21.15.40:0/3883763286 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3924050ea0 con 0x7f393c0b1460 2023-09-30T00:49:23.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.451+0000 7f39337fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:23.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.451+0000 7f39337fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:23.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.711+0000 7f3941bee700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:23.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.711+0000 7f3941bee700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:23.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.711+0000 7f3941bee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:23.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.711+0000 7f3941bee700 1 -- 172.21.15.40:0/3883763286 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f393c0a9960 con 0x7f393c0b1460 2023-09-30T00:49:23.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.716+0000 7f39337fe700 1 -- 172.21.15.40:0/3883763286 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f393c0a9960 con 0x7f393c0b1460 2023-09-30T00:49:23.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.716+0000 7f39337fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:23.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.716+0000 7f39337fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:23.717 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:23.717 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:23.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 10 librados: watch_flush enter 2023-09-30T00:49:23.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 10 librados: watch_flush exit 2023-09-30T00:49:23.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 20 client.4522.objecter shutdown clearing up homeless session... 2023-09-30T00:49:23.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 10 client.4522.objecter successfully canceled tick 2023-09-30T00:49:23.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 1 -- 172.21.15.40:0/3883763286 >> v2:172.21.15.135:6800/116426 conn(0x7f391c068890 msgr2=0x7f391c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:23.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 1 --2- 172.21.15.40:0/3883763286 >> v2:172.21.15.135:6800/116426 conn(0x7f391c068890 0x7f391c06ad40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f393c0b3fa0 tx=0x7f392c009250 comp rx=0 tx=0).stop 2023-09-30T00:49:23.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 10 monclient: shutdown 2023-09-30T00:49:23.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:23.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 1 -- 172.21.15.40:0/3883763286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f393c0b1460 msgr2=0x7f393c13e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:23.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 1 --2- 172.21.15.40:0/3883763286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f393c0b1460 0x7f393c13e440 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f392400c590 tx=0x7f392400c8a0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 1 -- 172.21.15.40:0/3883763286 shutdown_connections 2023-09-30T00:49:23.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 1 --2- 172.21.15.40:0/3883763286 >> v2:172.21.15.135:6800/116426 conn(0x7f391c068890 0x7f391c06ad40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 1 --2- 172.21.15.40:0/3883763286 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f393c0b1460 0x7f393c13e440 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 1 --2- 172.21.15.40:0/3883763286 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f393c13ac20 0x7f393c150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 1 --2- 172.21.15.40:0/3883763286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f393c0b1830 0x7f393c150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 1 -- 172.21.15.40:0/3883763286 >> 172.21.15.40:0/3883763286 conn(0x7f393c0631f0 msgr2=0x7f393c0afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:23.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 1 -- 172.21.15.40:0/3883763286 shutdown_connections 2023-09-30T00:49:23.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 1 -- 172.21.15.40:0/3883763286 wait complete. 2023-09-30T00:49:23.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:23.718+0000 7f3941bee700 1 librados: shutdown 2023-09-30T00:49:26.734 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:26.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.874+0000 7f3179f63700 10 monclient: get_monmap_and_config 2023-09-30T00:49:26.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.874+0000 7f3179f63700 10 monclient: build_initial_monmap 2023-09-30T00:49:26.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.874+0000 7f3179f63700 10 monclient: monmap: 2023-09-30T00:49:26.875 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:26.875 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:26.875 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:26.875278+0000 2023-09-30T00:49:26.875 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:26.875278+0000 2023-09-30T00:49:26.875 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:26.875 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:26.876 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:26.876 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:26.876 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:26.876 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:26.876 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:26.876 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 1 Processor -- start 2023-09-30T00:49:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 1 -- start start 2023-09-30T00:49:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 10 monclient: init 2023-09-30T00:49:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-09-30T00:49:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f31740b1460 0x7f31740b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 10 monclient(hunting): picked mon.noname-a con 0x7f31740b1460 addr v2:172.21.15.40:3300/0 2023-09-30T00:49:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31740a5fb0 0x7f3174141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 10 monclient(hunting): picked mon.noname-b con 0x7f31740a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31740af410 addr v1:172.21.15.135:6789/0 2023-09-30T00:49:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31740a71f0 con 0x7f31740af410 2023-09-30T00:49:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31740a74d0 con 0x7f31740b1460 2023-09-30T00:49:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31740a63c0 con 0x7f31740a5fb0 2023-09-30T00:49:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.875+0000 7f3179f63700 10 monclient(hunting): authenticate will time out at 5428.176765s 2023-09-30T00:49:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.876+0000 7f31737fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f31740b1460 0x7f31740b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.876+0000 7f3173fff700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f31740af410 0x7f31740af890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:51014/0 (socket says 172.21.15.40:51014) 2023-09-30T00:49:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.876+0000 7f3173fff700 1 -- 172.21.15.40:0/1407403101 learned_addr learned my addr 172.21.15.40:0/1407403101 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:49:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.876+0000 7f3172ffd700 1 --2- 172.21.15.40:0/1407403101 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31740a5fb0 0x7f3174141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.877+0000 7f31737fe700 10 monclient(hunting): get_auth_request con 0x7f31740b1460 auth_method 0 2023-09-30T00:49:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.877+0000 7f31737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.877+0000 7f31737fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.877+0000 7f31737fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.877+0000 7f3172ffd700 10 monclient(hunting): get_auth_request con 0x7f31740a5fb0 auth_method 0 2023-09-30T00:49:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.877+0000 7f3172ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.877+0000 7f3172ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.877+0000 7f3172ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.877+0000 7f31737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.877+0000 7f31737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.877+0000 7f31737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.877+0000 7f3172ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.877+0000 7f3172ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.877+0000 7f3172ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.880+0000 7f31727fc700 1 -- 172.21.15.40:0/1407403101 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177431726 0 0) 0x7f31740a71f0 con 0x7f31740af410 2023-09-30T00:49:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.880+0000 7f31727fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.880+0000 7f31727fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.880+0000 7f31727fc700 10 monclient(hunting): my global_id is 4525 2023-09-30T00:49:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.880+0000 7f31727fc700 1 -- 172.21.15.40:0/1407403101 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f315c003570 con 0x7f31740af410 2023-09-30T00:49:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.880+0000 7f3172ffd700 10 monclient(hunting): handle_auth_done global_id 4424 payload 274 2023-09-30T00:49:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.880+0000 7f3172ffd700 1 -- 172.21.15.40:0/1407403101 >> v2:172.21.15.40:3300/0 conn(0x7f31740b1460 msgr2=0x7f31740b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.880+0000 7f3172ffd700 1 --2- 172.21.15.40:0/1407403101 >> v2:172.21.15.40:3300/0 conn(0x7f31740b1460 0x7f31740b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.880+0000 7f3172ffd700 1 -- 172.21.15.40:0/1407403101 >> v1:172.21.15.135:6789/0 conn(0x7f31740af410 legacy=0x7f31740af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.880+0000 7f3172ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.880+0000 7f3172ffd700 1 monclient: found mon.noname-b 2023-09-30T00:49:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.880+0000 7f3172ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.880+0000 7f3172ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:49:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.880+0000 7f3172ffd700 1 -- 172.21.15.40:0/1407403101 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31740a56d0 con 0x7f31740a5fb0 2023-09-30T00:49:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.882+0000 7f3172ffd700 1 --2- 172.21.15.40:0/1407403101 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31740a5fb0 0x7f3174141730 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f31680098e0 tx=0x7f31680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97543b46d8b0de4b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.886+0000 7f31727fc700 1 -- 172.21.15.40:0/1407403101 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3168011630 con 0x7f31740a5fb0 2023-09-30T00:49:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.886+0000 7f31727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.886+0000 7f31727fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.886+0000 7f31727fc700 10 monclient: dump: 2023-09-30T00:49:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:26.889 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.886+0000 7f31727fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.886+0000 7f31727fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.886+0000 7f31727fc700 1 -- 172.21.15.40:0/1407403101 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3168011790 con 0x7f31740a5fb0 2023-09-30T00:49:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.886+0000 7f3179f63700 5 monclient: authenticate success, global_id 4424 2023-09-30T00:49:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.886+0000 7f3179f63700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.886+0000 7f31727fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.886+0000 7f3179f63700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.886+0000 7f3179f63700 4 set_mon_vals no callback set 2023-09-30T00:49:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.886+0000 7f31727fc700 1 -- 172.21.15.40:0/1407403101 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3168011b30 con 0x7f31740a5fb0 2023-09-30T00:49:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.886+0000 7f31727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.887+0000 7f31727fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.887+0000 7f31727fc700 10 monclient: dump: 2023-09-30T00:49:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:26.891 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.887+0000 7f3179f63700 10 monclient: shutdown 2023-09-30T00:49:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.887+0000 7f3179f63700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.887+0000 7f3179f63700 1 -- 172.21.15.40:0/1407403101 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31740a5fb0 msgr2=0x7f3174141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.887+0000 7f3179f63700 1 --2- 172.21.15.40:0/1407403101 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31740a5fb0 0x7f3174141730 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f31680098e0 tx=0x7f31680091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.887+0000 7f3179f63700 1 -- 172.21.15.40:0/1407403101 shutdown_connections 2023-09-30T00:49:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.887+0000 7f3179f63700 1 --2- 172.21.15.40:0/1407403101 >> v2:172.21.15.40:3300/0 conn(0x7f31740b1460 0x7f31740b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.887+0000 7f3179f63700 1 --2- 172.21.15.40:0/1407403101 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31740a5fb0 0x7f3174141730 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.887+0000 7f3179f63700 1 -- 172.21.15.40:0/1407403101 >> 172.21.15.40:0/1407403101 conn(0x7f31740631f0 msgr2=0x7f31740635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.887+0000 7f3179f63700 1 -- 172.21.15.40:0/1407403101 shutdown_connections 2023-09-30T00:49:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.887+0000 7f3179f63700 1 -- 172.21.15.40:0/1407403101 wait complete. 2023-09-30T00:49:26.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.888+0000 7f3179f63700 10 monclient: build_initial_monmap 2023-09-30T00:49:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.888+0000 7f3179f63700 10 monclient: monmap: 2023-09-30T00:49:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:26.895 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.888+0000 7f3179f63700 1 librados: starting msgr at 2023-09-30T00:49:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.888+0000 7f3179f63700 1 librados: starting objecter 2023-09-30T00:49:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 1 Processor -- start 2023-09-30T00:49:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 1 -- start start 2023-09-30T00:49:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 1 librados: setting wanted keys 2023-09-30T00:49:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 1 librados: calling monclient init 2023-09-30T00:49:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 10 monclient: init 2023-09-30T00:49:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:49:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31740a5fb0 0x7f317413ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 10 monclient(hunting): picked mon.noname-b con 0x7f31740a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31740af410 0x7f31740a7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 10 monclient(hunting): picked mon.noname-c con 0x7f31740af410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31740b1460 0x7f31741398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 10 monclient(hunting): picked mon.noname-a con 0x7f31740b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31740aa6b0 con 0x7f31740b1460 2023-09-30T00:49:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31740ab650 con 0x7f31740a5fb0 2023-09-30T00:49:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317413e300 con 0x7f31740af410 2023-09-30T00:49:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.889+0000 7f3179f63700 10 monclient(hunting): authenticate will time out at 5428.190716s 2023-09-30T00:49:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f3172ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31740af410 0x7f31740a7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f31737fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31740a5fb0 0x7f317413ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f3173fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31740b1460 0x7f31741398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f3173fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31740b1460 0x7f31741398c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45060/0 (socket says 172.21.15.40:45060) 2023-09-30T00:49:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f3173fff700 1 -- 172.21.15.40:0/692730822 learned_addr learned my addr 172.21.15.40:0/692730822 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f3173fff700 10 monclient(hunting): get_auth_request con 0x7f31740b1460 auth_method 0 2023-09-30T00:49:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f3173fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f3173fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f3173fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f3172ffd700 10 monclient(hunting): get_auth_request con 0x7f31740af410 auth_method 0 2023-09-30T00:49:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f3172ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f3172ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f3172ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f31737fe700 10 monclient(hunting): get_auth_request con 0x7f31740a5fb0 auth_method 0 2023-09-30T00:49:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f31737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f31737fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.890+0000 7f31737fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.891+0000 7f3172ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.891+0000 7f3172ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.891+0000 7f3172ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.891+0000 7f3173fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.891+0000 7f3173fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.891+0000 7f3173fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.891+0000 7f31737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.891+0000 7f31737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.891+0000 7f31737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.895+0000 7f3172ffd700 10 monclient(hunting): handle_auth_done global_id 4427 payload 824 2023-09-30T00:49:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.895+0000 7f3172ffd700 1 -- 172.21.15.40:0/692730822 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31740a5fb0 msgr2=0x7f317413ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.895+0000 7f3172ffd700 1 --2- 172.21.15.40:0/692730822 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31740a5fb0 0x7f317413ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.895+0000 7f3172ffd700 1 -- 172.21.15.40:0/692730822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31740b1460 msgr2=0x7f31741398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.895+0000 7f3172ffd700 1 --2- 172.21.15.40:0/692730822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31740b1460 0x7f31741398c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.895+0000 7f3172ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.895+0000 7f3172ffd700 1 monclient: found mon.noname-c 2023-09-30T00:49:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.895+0000 7f3172ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.895+0000 7f3172ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:49:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.895+0000 7f3172ffd700 1 -- 172.21.15.40:0/692730822 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3168009420 con 0x7f31740af410 2023-09-30T00:49:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.897+0000 7f3172ffd700 1 --2- 172.21.15.40:0/692730822 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31740af410 0x7f31740a7e50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3168016000 tx=0x7f3168017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.897+0000 7f315b7fe700 10 client.?.objecter ms_handle_connect 0x7f31740af410 2023-09-30T00:49:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.897+0000 7f315b7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f315b7fe700 1 -- 172.21.15.40:0/692730822 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3168011b10 con 0x7f31740af410 2023-09-30T00:49:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f315b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f315b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f315b7fe700 10 monclient: dump: 2023-09-30T00:49:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:26.904 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f315b7fe700 10 monclient: _finish_auth 0 2023-09-30T00:49:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f315b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f3179f63700 5 monclient: authenticate success, global_id 4427 2023-09-30T00:49:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f315b7fe700 1 -- 172.21.15.40:0/692730822 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3168000c90 con 0x7f31740af410 2023-09-30T00:49:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f3179f63700 10 monclient: _renew_subs 2023-09-30T00:49:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f3179f63700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f3179f63700 1 -- 172.21.15.40:0/692730822 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31740a8390 con 0x7f31740af410 2023-09-30T00:49:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f315b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f315b7fe700 1 -- 172.21.15.40:0/692730822 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3168025640 con 0x7f31740af410 2023-09-30T00:49:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f315b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f315b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f315b7fe700 10 monclient: dump: 2023-09-30T00:49:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:26.906 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f3171ffb700 4 set_mon_vals no callback set 2023-09-30T00:49:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f3179f63700 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f3179f63700 10 monclient: _renew_subs 2023-09-30T00:49:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f3179f63700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f3179f63700 1 -- 172.21.15.40:0/692730822 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3174139e00 con 0x7f31740af410 2023-09-30T00:49:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.902+0000 7f3179f63700 1 librados: init done 2023-09-30T00:49:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.903+0000 7f3179f63700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.903+0000 7f3179f63700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.903+0000 7f3179f63700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:26.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.903+0000 7f3179f63700 1 -- 172.21.15.40:0/692730822 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31740a9080 con 0x7f31740af410 2023-09-30T00:49:26.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.923+0000 7f315b7fe700 1 -- 172.21.15.40:0/692730822 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f316802e450 con 0x7f31740af410 2023-09-30T00:49:26.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.923+0000 7f315b7fe700 1 --2- 172.21.15.40:0/692730822 >> v2:172.21.15.135:6800/116426 conn(0x7f3160068960 0x7f316006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:26.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.924+0000 7f31737fe700 1 --2- 172.21.15.40:0/692730822 >> v2:172.21.15.135:6800/116426 conn(0x7f3160068960 0x7f316006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:26.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.924+0000 7f31737fe700 10 monclient: get_auth_request con 0x7f3160068960 auth_method 0 2023-09-30T00:49:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.924+0000 7f315b7fe700 10 client.4427.objecter ms_handle_connect 0x7f3160068960 2023-09-30T00:49:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:26.924+0000 7f31737fe700 1 --2- 172.21.15.40:0/692730822 >> v2:172.21.15.135:6800/116426 conn(0x7f3160068960 0x7f316006ae10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3164009d10 tx=0x7f31640091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.105+0000 7f315b7fe700 1 -- 172.21.15.40:0/692730822 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3168057380 con 0x7f31740af410 2023-09-30T00:49:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.105+0000 7f315b7fe700 10 client.4427.objecter ms_dispatch 0x7f3174001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.105+0000 7f315b7fe700 3 client.4427.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.105+0000 7f315b7fe700 3 client.4427.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:27.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.105+0000 7f315b7fe700 20 client.4427.objecter dump_active .. 0 homeless 2023-09-30T00:49:27.201 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.200+0000 7f315b7fe700 1 -- 172.21.15.40:0/692730822 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3168052fc0 con 0x7f31740af410 2023-09-30T00:49:27.201 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.200+0000 7f315b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:27.201 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.200+0000 7f315b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:27.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.456+0000 7f3179f63700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:27.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.456+0000 7f3179f63700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:27.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.456+0000 7f3179f63700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:27.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.456+0000 7f3179f63700 1 -- 172.21.15.40:0/692730822 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f31740a9080 con 0x7f31740af410 2023-09-30T00:49:27.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.461+0000 7f315b7fe700 1 -- 172.21.15.40:0/692730822 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f31740a9080 con 0x7f31740af410 2023-09-30T00:49:27.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.461+0000 7f315b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:27.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.461+0000 7f315b7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:27.462 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:27.462 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:27.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 10 librados: watch_flush enter 2023-09-30T00:49:27.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 10 librados: watch_flush exit 2023-09-30T00:49:27.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 20 client.4427.objecter shutdown clearing up homeless session... 2023-09-30T00:49:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 10 client.4427.objecter successfully canceled tick 2023-09-30T00:49:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 1 -- 172.21.15.40:0/692730822 >> v2:172.21.15.135:6800/116426 conn(0x7f3160068960 msgr2=0x7f316006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 1 --2- 172.21.15.40:0/692730822 >> v2:172.21.15.135:6800/116426 conn(0x7f3160068960 0x7f316006ae10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3164009d10 tx=0x7f31640091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 10 monclient: shutdown 2023-09-30T00:49:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 1 -- 172.21.15.40:0/692730822 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31740af410 msgr2=0x7f31740a7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 1 --2- 172.21.15.40:0/692730822 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31740af410 0x7f31740a7e50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3168016000 tx=0x7f3168017000 comp rx=0 tx=0).stop 2023-09-30T00:49:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 1 -- 172.21.15.40:0/692730822 shutdown_connections 2023-09-30T00:49:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 1 --2- 172.21.15.40:0/692730822 >> v2:172.21.15.135:6800/116426 conn(0x7f3160068960 0x7f316006ae10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 1 --2- 172.21.15.40:0/692730822 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f31740a5fb0 0x7f317413ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 1 --2- 172.21.15.40:0/692730822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31740b1460 0x7f31741398c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 1 --2- 172.21.15.40:0/692730822 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f31740af410 0x7f31740a7e50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:27.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 1 -- 172.21.15.40:0/692730822 >> 172.21.15.40:0/692730822 conn(0x7f31740631f0 msgr2=0x7f3174144a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:27.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 1 -- 172.21.15.40:0/692730822 shutdown_connections 2023-09-30T00:49:27.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 1 -- 172.21.15.40:0/692730822 wait complete. 2023-09-30T00:49:27.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:27.463+0000 7f3179f63700 1 librados: shutdown 2023-09-30T00:49:27.995 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:03:03.706 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:28.059 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:03:03.770 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:28.095 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:03:03.806 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:28.115 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:03.802 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:28.117 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:03:03.829 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:28.121 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:03:03.833 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:28.124 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:03.810 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:28.149 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:03:03.860 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:28.153 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:03.839 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:30.479 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:30.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.617+0000 7f2e3c226700 10 monclient: get_monmap_and_config 2023-09-30T00:49:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.617+0000 7f2e3c226700 10 monclient: build_initial_monmap 2023-09-30T00:49:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.617+0000 7f2e3c226700 10 monclient: monmap: 2023-09-30T00:49:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:30.618372+0000 2023-09-30T00:49:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:30.618372+0000 2023-09-30T00:49:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:30.619 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:30.619 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:30.619 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:30.619 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:30.619 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.617+0000 7f2e3c226700 1 Processor -- start 2023-09-30T00:49:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 1 -- start start 2023-09-30T00:49:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 10 monclient: init 2023-09-30T00:49:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-09-30T00:49:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e340b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:49:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2e340b1830 0x7f2e340a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 10 monclient(hunting): picked mon.noname-c con 0x7f2e340b1830 addr v2:172.21.15.135:3300/0 2023-09-30T00:49:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2e3413ac20 0x7f2e3413b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 10 monclient(hunting): picked mon.noname-a con 0x7f2e3413ac20 addr v2:172.21.15.40:3300/0 2023-09-30T00:49:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e340a5b20 con 0x7f2e340b1460 2023-09-30T00:49:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e34141e40 con 0x7f2e3413ac20 2023-09-30T00:49:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e340a64f0 con 0x7f2e340b1830 2023-09-30T00:49:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.618+0000 7f2e3c226700 10 monclient(hunting): authenticate will time out at 5431.919950s 2023-09-30T00:49:30.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.661+0000 7f2e397c1700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2e340b1830 0x7f2e340a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:30.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.661+0000 7f2e397c1700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f2e340b1830 0x7f2e340a5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:50700/0 (socket says 172.21.15.40:50700) 2023-09-30T00:49:30.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.661+0000 7f2e397c1700 1 -- 172.21.15.40:0/1649605475 learned_addr learned my addr 172.21.15.40:0/1649605475 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:30.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.661+0000 7f2e397c1700 10 monclient(hunting): get_auth_request con 0x7f2e340b1830 auth_method 0 2023-09-30T00:49:30.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.661+0000 7f2e397c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:30.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.661+0000 7f2e397c1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:30.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.661+0000 7f2e397c1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:30.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.662+0000 7f2e397c1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:30.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.662+0000 7f2e397c1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:30.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.662+0000 7f2e397c1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:30.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.664+0000 7f2e3a7c3700 1 --2- 172.21.15.40:0/1649605475 >> v2:172.21.15.40:3300/0 conn(0x7f2e3413ac20 0x7f2e3413b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:30.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.665+0000 7f2e397c1700 10 monclient(hunting): handle_auth_done global_id 4531 payload 274 2023-09-30T00:49:30.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.665+0000 7f2e397c1700 1 -- 172.21.15.40:0/1649605475 >> v2:172.21.15.40:3300/0 conn(0x7f2e3413ac20 msgr2=0x7f2e3413b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:30.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.665+0000 7f2e397c1700 1 --2- 172.21.15.40:0/1649605475 >> v2:172.21.15.40:3300/0 conn(0x7f2e3413ac20 0x7f2e3413b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:30.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.665+0000 7f2e397c1700 1 -- 172.21.15.40:0/1649605475 >> v1:172.21.15.135:6789/0 conn(0x7f2e340b1460 legacy=0x7f2e34141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:30.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.665+0000 7f2e397c1700 10 monclient: _finish_hunting 0 2023-09-30T00:49:30.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.665+0000 7f2e397c1700 1 monclient: found mon.noname-c 2023-09-30T00:49:30.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.665+0000 7f2e397c1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:30.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.665+0000 7f2e397c1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:49:30.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.665+0000 7f2e397c1700 1 -- 172.21.15.40:0/1649605475 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e340a71f0 con 0x7f2e340b1830 2023-09-30T00:49:30.668 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.668+0000 7f2e397c1700 1 --2- 172.21.15.40:0/1649605475 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e340b1830 0x7f2e340a5fb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2e240099b0 tx=0x7f2e24009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=238ffe60f6cbd2d1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:30.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e38fc0700 1 -- 172.21.15.40:0/1649605475 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2400b7e0 con 0x7f2e340b1830 2023-09-30T00:49:30.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e38fc0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:30.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e38fc0700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:30.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e38fc0700 10 monclient: dump: 2023-09-30T00:49:30.679 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:30.679 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:30.679 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:30.679 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:30.679 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:30.679 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:30.679 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:30.679 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:30.679 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:30.679 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e38fc0700 10 monclient: _finish_auth 0 2023-09-30T00:49:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e38fc0700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e38fc0700 1 -- 172.21.15.40:0/1649605475 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2400b940 con 0x7f2e340b1830 2023-09-30T00:49:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e3c226700 5 monclient: authenticate success, global_id 4531 2023-09-30T00:49:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e3c226700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e38fc0700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e3c226700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e38fc0700 1 -- 172.21.15.40:0/1649605475 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2400bce0 con 0x7f2e340b1830 2023-09-30T00:49:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e3c226700 4 set_mon_vals no callback set 2023-09-30T00:49:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e38fc0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e38fc0700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:49:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e38fc0700 10 monclient: dump: 2023-09-30T00:49:30.681 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:30.681 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:30.681 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:30.681 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:30.681 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:30.681 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:30.681 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:30.681 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:30.681 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:30.681 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:30.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e3c226700 10 monclient: shutdown 2023-09-30T00:49:30.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e3c226700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:30.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e3c226700 1 -- 172.21.15.40:0/1649605475 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e340b1830 msgr2=0x7f2e340a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:30.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e3c226700 1 --2- 172.21.15.40:0/1649605475 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e340b1830 0x7f2e340a5fb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2e240099b0 tx=0x7f2e24009250 comp rx=0 tx=0).stop 2023-09-30T00:49:30.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e3c226700 1 -- 172.21.15.40:0/1649605475 shutdown_connections 2023-09-30T00:49:30.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e3c226700 1 --2- 172.21.15.40:0/1649605475 >> v2:172.21.15.40:3300/0 conn(0x7f2e3413ac20 0x7f2e3413b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:30.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e3c226700 1 --2- 172.21.15.40:0/1649605475 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e340b1830 0x7f2e340a5fb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.678+0000 7f2e3c226700 1 -- 172.21.15.40:0/1649605475 >> 172.21.15.40:0/1649605475 conn(0x7f2e340631f0 msgr2=0x7f2e340635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.679+0000 7f2e3c226700 1 -- 172.21.15.40:0/1649605475 shutdown_connections 2023-09-30T00:49:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.679+0000 7f2e3c226700 1 -- 172.21.15.40:0/1649605475 wait complete. 2023-09-30T00:49:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.679+0000 7f2e3c226700 10 monclient: build_initial_monmap 2023-09-30T00:49:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.679+0000 7f2e3c226700 10 monclient: monmap: 2023-09-30T00:49:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:30.685 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:30.686 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:30.686 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:30.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.680+0000 7f2e3c226700 1 librados: starting msgr at 2023-09-30T00:49:30.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.680+0000 7f2e3c226700 1 librados: starting objecter 2023-09-30T00:49:30.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.680+0000 7f2e3c226700 1 Processor -- start 2023-09-30T00:49:30.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.680+0000 7f2e3c226700 1 -- start start 2023-09-30T00:49:30.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.680+0000 7f2e3c226700 1 librados: setting wanted keys 2023-09-30T00:49:30.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.680+0000 7f2e3c226700 1 librados: calling monclient init 2023-09-30T00:49:30.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.680+0000 7f2e3c226700 10 monclient: init 2023-09-30T00:49:30.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:30.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:49:30.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e340b1460 0x7f2e3413dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:30.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 10 monclient(hunting): picked mon.noname-b con 0x7f2e340b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:30.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e340b1830 0x7f2e340a79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:30.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 10 monclient(hunting): picked mon.noname-a con 0x7f2e340b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:30.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e3413ac20 0x7f2e340a9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:30.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 10 monclient(hunting): picked mon.noname-c con 0x7f2e3413ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:30.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:30.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e34139ab0 con 0x7f2e340b1830 2023-09-30T00:49:30.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:30.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340a7f40 con 0x7f2e340b1460 2023-09-30T00:49:30.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:30.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340a9a80 con 0x7f2e3413ac20 2023-09-30T00:49:30.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:30.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3c226700 10 monclient(hunting): authenticate will time out at 5431.982267s 2023-09-30T00:49:30.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.681+0000 7f2e3a7c3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e3413ac20 0x7f2e340a9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:30.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.682+0000 7f2e39fc2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e340b1460 0x7f2e3413dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:30.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.682+0000 7f2e39fc2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e340b1460 0x7f2e3413dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:50704/0 (socket says 172.21.15.40:50704) 2023-09-30T00:49:30.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.682+0000 7f2e39fc2700 1 -- 172.21.15.40:0/4125005106 learned_addr learned my addr 172.21.15.40:0/4125005106 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:30.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.682+0000 7f2e39fc2700 10 monclient(hunting): get_auth_request con 0x7f2e340b1460 auth_method 0 2023-09-30T00:49:30.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.682+0000 7f2e39fc2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:30.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.682+0000 7f2e39fc2700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:30.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.682+0000 7f2e39fc2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:30.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.682+0000 7f2e3a7c3700 10 monclient(hunting): get_auth_request con 0x7f2e3413ac20 auth_method 0 2023-09-30T00:49:30.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.682+0000 7f2e3a7c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:30.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.682+0000 7f2e3a7c3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:30.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.682+0000 7f2e3a7c3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:30.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.683+0000 7f2e39fc2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:30.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.683+0000 7f2e39fc2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:30.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.683+0000 7f2e39fc2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:30.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.683+0000 7f2e3a7c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:30.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.683+0000 7f2e3a7c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:30.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.683+0000 7f2e3a7c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:30.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.684+0000 7f2e397c1700 1 --2- 172.21.15.40:0/4125005106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e340b1830 0x7f2e340a79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:30.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.685+0000 7f2e397c1700 10 monclient(hunting): get_auth_request con 0x7f2e340b1830 auth_method 0 2023-09-30T00:49:30.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.685+0000 7f2e397c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:30.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.685+0000 7f2e397c1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:30.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.685+0000 7f2e397c1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:30.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.685+0000 7f2e397c1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:30.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.685+0000 7f2e397c1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:30.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.685+0000 7f2e397c1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:30.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.686+0000 7f2e39fc2700 10 monclient(hunting): handle_auth_done global_id 4534 payload 824 2023-09-30T00:49:30.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.686+0000 7f2e39fc2700 1 -- 172.21.15.40:0/4125005106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e3413ac20 msgr2=0x7f2e340a9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:30.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.686+0000 7f2e39fc2700 1 --2- 172.21.15.40:0/4125005106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e3413ac20 0x7f2e340a9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:30.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.686+0000 7f2e39fc2700 1 -- 172.21.15.40:0/4125005106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e340b1830 msgr2=0x7f2e340a79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:30.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.686+0000 7f2e39fc2700 1 --2- 172.21.15.40:0/4125005106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e340b1830 0x7f2e340a79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:30.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.686+0000 7f2e39fc2700 10 monclient: _finish_hunting 0 2023-09-30T00:49:30.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.686+0000 7f2e39fc2700 1 monclient: found mon.noname-b 2023-09-30T00:49:30.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.686+0000 7f2e39fc2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:30.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.686+0000 7f2e39fc2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:49:30.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.686+0000 7f2e39fc2700 1 -- 172.21.15.40:0/4125005106 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e240094f0 con 0x7f2e340b1460 2023-09-30T00:49:30.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.689+0000 7f2e39fc2700 1 --2- 172.21.15.40:0/4125005106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e340b1460 0x7f2e3413dfc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000c5d0 tx=0x7f2e30007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:30.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.689+0000 7f2e29ffb700 10 client.?.objecter ms_handle_connect 0x7f2e340b1460 2023-09-30T00:49:30.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.689+0000 7f2e29ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:30.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.692+0000 7f2e29ffb700 1 -- 172.21.15.40:0/4125005106 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e30011590 con 0x7f2e340b1460 2023-09-30T00:49:30.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e29ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:30.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e29ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:30.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e29ffb700 10 monclient: dump: 2023-09-30T00:49:30.694 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:30.694 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:30.694 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:30.694 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:30.694 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:30.694 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:30.694 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:30.694 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:30.694 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:30.694 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:30.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e29ffb700 10 monclient: _finish_auth 0 2023-09-30T00:49:30.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e29ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:30.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e29ffb700 1 -- 172.21.15.40:0/4125005106 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e30011710 con 0x7f2e340b1460 2023-09-30T00:49:30.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e3c226700 5 monclient: authenticate success, global_id 4534 2023-09-30T00:49:30.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e29ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:30.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e29ffb700 1 -- 172.21.15.40:0/4125005106 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e30011ad0 con 0x7f2e340b1460 2023-09-30T00:49:30.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e29ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:30.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e29ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:49:30.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e29ffb700 10 monclient: dump: 2023-09-30T00:49:30.695 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:30.695 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:30.695 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:30.695 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:30.696 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:30.696 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:30.696 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:30.696 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:30.696 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:30.696 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:30.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e2bfff700 4 set_mon_vals no callback set 2023-09-30T00:49:30.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e3c226700 10 monclient: _renew_subs 2023-09-30T00:49:30.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e3c226700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:30.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.693+0000 7f2e3c226700 1 -- 172.21.15.40:0/4125005106 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3413ec80 con 0x7f2e340b1460 2023-09-30T00:49:30.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.694+0000 7f2e3c226700 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:30.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.694+0000 7f2e3c226700 10 monclient: _renew_subs 2023-09-30T00:49:30.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.694+0000 7f2e3c226700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:30.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.694+0000 7f2e3c226700 1 -- 172.21.15.40:0/4125005106 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e340a8f80 con 0x7f2e340b1460 2023-09-30T00:49:30.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.694+0000 7f2e3c226700 1 librados: init done 2023-09-30T00:49:30.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.695+0000 7f2e3c226700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:30.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.695+0000 7f2e3c226700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:30.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.695+0000 7f2e3c226700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:30.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.695+0000 7f2e3c226700 1 -- 172.21.15.40:0/4125005106 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3406f560 con 0x7f2e340b1460 2023-09-30T00:49:30.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.714+0000 7f2e29ffb700 1 -- 172.21.15.40:0/4125005106 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f2e3001c460 con 0x7f2e340b1460 2023-09-30T00:49:30.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.714+0000 7f2e29ffb700 1 --2- 172.21.15.40:0/4125005106 >> v2:172.21.15.135:6800/116426 conn(0x7f2e18068890 0x7f2e1806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:30.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.715+0000 7f2e397c1700 1 --2- 172.21.15.40:0/4125005106 >> v2:172.21.15.135:6800/116426 conn(0x7f2e18068890 0x7f2e1806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:30.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.715+0000 7f2e397c1700 10 monclient: get_auth_request con 0x7f2e18068890 auth_method 0 2023-09-30T00:49:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.715+0000 7f2e397c1700 1 --2- 172.21.15.40:0/4125005106 >> v2:172.21.15.135:6800/116426 conn(0x7f2e18068890 0x7f2e1806ad40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2e240030c0 tx=0x7f2e24016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.715+0000 7f2e29ffb700 10 client.4534.objecter ms_handle_connect 0x7f2e18068890 2023-09-30T00:49:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.719+0000 7f2e29ffb700 1 -- 172.21.15.40:0/4125005106 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f2e30055c30 con 0x7f2e340b1460 2023-09-30T00:49:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.719+0000 7f2e29ffb700 10 client.4534.objecter ms_dispatch 0x7f2e34001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.719+0000 7f2e29ffb700 3 client.4534.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.719+0000 7f2e29ffb700 3 client.4534.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.719+0000 7f2e29ffb700 20 client.4534.objecter dump_active .. 0 homeless 2023-09-30T00:49:30.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.964+0000 7f2e29ffb700 1 -- 172.21.15.40:0/4125005106 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2e30051f70 con 0x7f2e340b1460 2023-09-30T00:49:30.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.964+0000 7f2e29ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:30.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:30.964+0000 7f2e29ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:31.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.220+0000 7f2e3c226700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:31.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.220+0000 7f2e3c226700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:31.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.220+0000 7f2e3c226700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:31.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.220+0000 7f2e3c226700 1 -- 172.21.15.40:0/4125005106 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2e34064a00 con 0x7f2e340b1460 2023-09-30T00:49:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.226+0000 7f2e29ffb700 1 -- 172.21.15.40:0/4125005106 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2e30055590 con 0x7f2e340b1460 2023-09-30T00:49:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.226+0000 7f2e29ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.226+0000 7f2e29ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:31.226 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:31.227 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:31.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.227+0000 7f2e3c226700 10 librados: watch_flush enter 2023-09-30T00:49:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.227+0000 7f2e3c226700 10 librados: watch_flush exit 2023-09-30T00:49:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.227+0000 7f2e3c226700 20 client.4534.objecter shutdown clearing up homeless session... 2023-09-30T00:49:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.227+0000 7f2e3c226700 10 client.4534.objecter successfully canceled tick 2023-09-30T00:49:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 1 -- 172.21.15.40:0/4125005106 >> v2:172.21.15.135:6800/116426 conn(0x7f2e18068890 msgr2=0x7f2e1806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 1 --2- 172.21.15.40:0/4125005106 >> v2:172.21.15.135:6800/116426 conn(0x7f2e18068890 0x7f2e1806ad40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2e240030c0 tx=0x7f2e24016000 comp rx=0 tx=0).stop 2023-09-30T00:49:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 10 monclient: shutdown 2023-09-30T00:49:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 1 -- 172.21.15.40:0/4125005106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e340b1460 msgr2=0x7f2e3413dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 1 --2- 172.21.15.40:0/4125005106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e340b1460 0x7f2e3413dfc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000c5d0 tx=0x7f2e30007f60 comp rx=0 tx=0).stop 2023-09-30T00:49:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 1 -- 172.21.15.40:0/4125005106 shutdown_connections 2023-09-30T00:49:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 1 --2- 172.21.15.40:0/4125005106 >> v2:172.21.15.135:6800/116426 conn(0x7f2e18068890 0x7f2e1806ad40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 1 --2- 172.21.15.40:0/4125005106 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f2e340b1460 0x7f2e3413dfc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 1 --2- 172.21.15.40:0/4125005106 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f2e3413ac20 0x7f2e340a9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 1 --2- 172.21.15.40:0/4125005106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e340b1830 0x7f2e340a79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 1 -- 172.21.15.40:0/4125005106 >> 172.21.15.40:0/4125005106 conn(0x7f2e340631f0 msgr2=0x7f2e34142870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 1 -- 172.21.15.40:0/4125005106 shutdown_connections 2023-09-30T00:49:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 1 -- 172.21.15.40:0/4125005106 wait complete. 2023-09-30T00:49:31.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:31.228+0000 7f2e3c226700 1 librados: shutdown 2023-09-30T00:49:34.243 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:34.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.377+0000 7f88161b2700 10 monclient: get_monmap_and_config 2023-09-30T00:49:34.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.377+0000 7f88161b2700 10 monclient: build_initial_monmap 2023-09-30T00:49:34.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.377+0000 7f88161b2700 10 monclient: monmap: 2023-09-30T00:49:34.377 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:34.377 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:34.378 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:34.377635+0000 2023-09-30T00:49:34.378 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:34.377635+0000 2023-09-30T00:49:34.378 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:34.378 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:34.378 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:34.378 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:34.378 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:34.378 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:34.378 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:34.378 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:34.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.378+0000 7f88161b2700 1 Processor -- start 2023-09-30T00:49:34.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.378+0000 7f88161b2700 1 -- start start 2023-09-30T00:49:34.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.378+0000 7f88161b2700 10 monclient: init 2023-09-30T00:49:34.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.379+0000 7f88161b2700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.379+0000 7f88161b2700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-09-30T00:49:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.379+0000 7f88161b2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88100b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:49:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.379+0000 7f88161b2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88100b1830 addr v1:172.21.15.135:6789/0 2023-09-30T00:49:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.379+0000 7f88161b2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f88100a5fb0 0x7f881013b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.379+0000 7f88161b2700 10 monclient(hunting): picked mon.noname-a con 0x7f88100a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:49:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.379+0000 7f88161b2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100a5b20 con 0x7f88100b1460 2023-09-30T00:49:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.379+0000 7f88161b2700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100a71f0 con 0x7f88100b1830 2023-09-30T00:49:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.379+0000 7f88161b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.379+0000 7f88161b2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8810141e40 con 0x7f88100a5fb0 2023-09-30T00:49:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.379+0000 7f88161b2700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.379+0000 7f88161b2700 10 monclient(hunting): authenticate will time out at 5435.679712s 2023-09-30T00:49:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.380+0000 7f880ffff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f88100a5fb0 0x7f881013b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:34.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.380+0000 7f880effd700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f88100b1830 0x7f881013ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:45660/0 (socket says 172.21.15.40:45660) 2023-09-30T00:49:34.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.380+0000 7f880effd700 1 -- 172.21.15.40:0/179647697 learned_addr learned my addr 172.21.15.40:0/179647697 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:49:34.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.380+0000 7f880ffff700 10 monclient(hunting): get_auth_request con 0x7f88100a5fb0 auth_method 0 2023-09-30T00:49:34.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.380+0000 7f880ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:34.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.380+0000 7f880ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:34.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.380+0000 7f880ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:34.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.381+0000 7f880ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:34.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.381+0000 7f880ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:34.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.381+0000 7f880ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:34.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.383+0000 7f880e7fc700 1 -- 172.21.15.40:0/179647697 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172793903 0 0) 0x7f88100a71f0 con 0x7f88100b1830 2023-09-30T00:49:34.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.383+0000 7f880e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:34.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.383+0000 7f880e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:34.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.383+0000 7f880e7fc700 10 monclient(hunting): my global_id is 4537 2023-09-30T00:49:34.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.383+0000 7f880e7fc700 1 -- 172.21.15.40:0/179647697 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f8003570 con 0x7f88100b1830 2023-09-30T00:49:34.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.384+0000 7f880ffff700 10 monclient(hunting): handle_auth_done global_id 4485 payload 274 2023-09-30T00:49:34.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.384+0000 7f880ffff700 1 -- 172.21.15.40:0/179647697 >> v1:172.21.15.135:6789/0 conn(0x7f88100b1830 legacy=0x7f881013ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:34.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.384+0000 7f880ffff700 1 -- 172.21.15.40:0/179647697 >> v1:172.21.15.40:6789/0 conn(0x7f88100b1460 legacy=0x7f8810141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:34.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.384+0000 7f880ffff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:34.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.384+0000 7f880ffff700 1 monclient: found mon.noname-a 2023-09-30T00:49:34.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.384+0000 7f880ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:34.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.384+0000 7f880ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:49:34.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.384+0000 7f880ffff700 1 -- 172.21.15.40:0/179647697 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88100a74d0 con 0x7f88100a5fb0 2023-09-30T00:49:34.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.387+0000 7f880ffff700 1 --2- 172.21.15.40:0/179647697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88100a5fb0 0x7f881013b330 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8808009e90 tx=0x7f8808009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82fced614ac0591 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:34.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f880e7fc700 1 -- 172.21.15.40:0/179647697 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880800e070 con 0x7f88100a5fb0 2023-09-30T00:49:34.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f880e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:34.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f880e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:34.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f880e7fc700 10 monclient: dump: 2023-09-30T00:49:34.393 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:34.394 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:34.394 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:34.394 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:34.394 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:34.394 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:34.394 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:34.394 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:34.394 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:34.394 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:34.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f880e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:34.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f880e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:34.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f880e7fc700 1 -- 172.21.15.40:0/179647697 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880800bbb0 con 0x7f88100a5fb0 2023-09-30T00:49:34.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f880e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:34.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f880e7fc700 1 -- 172.21.15.40:0/179647697 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808012430 con 0x7f88100a5fb0 2023-09-30T00:49:34.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f88161b2700 5 monclient: authenticate success, global_id 4485 2023-09-30T00:49:34.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f88161b2700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:34.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f880e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:34.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f88161b2700 4 set_mon_vals no callback set 2023-09-30T00:49:34.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f880e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:34.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.393+0000 7f880e7fc700 10 monclient: dump: 2023-09-30T00:49:34.395 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:34.395 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:34.395 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:34.396 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:34.396 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:34.396 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:34.396 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:34.396 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:34.396 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:34.396 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:34.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.394+0000 7f88161b2700 10 monclient: shutdown 2023-09-30T00:49:34.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.394+0000 7f88161b2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:34.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.394+0000 7f88161b2700 1 -- 172.21.15.40:0/179647697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88100a5fb0 msgr2=0x7f881013b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:34.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.394+0000 7f88161b2700 1 --2- 172.21.15.40:0/179647697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88100a5fb0 0x7f881013b330 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8808009e90 tx=0x7f8808009780 comp rx=0 tx=0).stop 2023-09-30T00:49:34.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.394+0000 7f88161b2700 1 -- 172.21.15.40:0/179647697 shutdown_connections 2023-09-30T00:49:34.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.394+0000 7f88161b2700 1 --2- 172.21.15.40:0/179647697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88100a5fb0 0x7f881013b330 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:34.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.394+0000 7f88161b2700 1 -- 172.21.15.40:0/179647697 >> 172.21.15.40:0/179647697 conn(0x7f88100631f0 msgr2=0x7f88100635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:34.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.394+0000 7f88161b2700 1 -- 172.21.15.40:0/179647697 shutdown_connections 2023-09-30T00:49:34.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.394+0000 7f88161b2700 1 -- 172.21.15.40:0/179647697 wait complete. 2023-09-30T00:49:34.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.395+0000 7f88161b2700 10 monclient: build_initial_monmap 2023-09-30T00:49:34.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.395+0000 7f88161b2700 10 monclient: monmap: 2023-09-30T00:49:34.400 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:34.400 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:34.400 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:34.401 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:34.401 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:34.401 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:34.401 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:34.401 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:34.401 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:34.401 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:34.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.395+0000 7f88161b2700 1 librados: starting msgr at 2023-09-30T00:49:34.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.395+0000 7f88161b2700 1 librados: starting objecter 2023-09-30T00:49:34.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.395+0000 7f88161b2700 1 Processor -- start 2023-09-30T00:49:34.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.395+0000 7f88161b2700 1 -- start start 2023-09-30T00:49:34.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.395+0000 7f88161b2700 1 librados: setting wanted keys 2023-09-30T00:49:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.395+0000 7f88161b2700 1 librados: calling monclient init 2023-09-30T00:49:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.395+0000 7f88161b2700 10 monclient: init 2023-09-30T00:49:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:49:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88100a5fb0 0x7f881013e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 10 monclient(hunting): picked mon.noname-c con 0x7f88100a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88100b1460 0x7f88100a9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 10 monclient(hunting): picked mon.noname-a con 0x7f88100b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88100b1830 0x7f88100aa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 10 monclient(hunting): picked mon.noname-b con 0x7f88100b1830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100a9ad0 con 0x7f88100b1460 2023-09-30T00:49:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100ab7a0 con 0x7f88100b1830 2023-09-30T00:49:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88100aba80 con 0x7f88100a5fb0 2023-09-30T00:49:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.396+0000 7f88161b2700 10 monclient(hunting): authenticate will time out at 5435.696620s 2023-09-30T00:49:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.397+0000 7f880f7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88100a5fb0 0x7f881013e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.397+0000 7f880effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88100b1460 0x7f88100a9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.397+0000 7f880ffff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88100b1830 0x7f88100aa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.397+0000 7f880f7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88100a5fb0 0x7f881013e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:59522/0 (socket says 172.21.15.40:59522) 2023-09-30T00:49:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.397+0000 7f880f7fe700 1 -- 172.21.15.40:0/2893951725 learned_addr learned my addr 172.21.15.40:0/2893951725 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.397+0000 7f880f7fe700 10 monclient(hunting): get_auth_request con 0x7f88100a5fb0 auth_method 0 2023-09-30T00:49:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.397+0000 7f880f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.397+0000 7f880f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.397+0000 7f880f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.397+0000 7f880effd700 10 monclient(hunting): get_auth_request con 0x7f88100b1460 auth_method 0 2023-09-30T00:49:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.397+0000 7f880effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.397+0000 7f880effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.397+0000 7f880effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.398+0000 7f880ffff700 10 monclient(hunting): get_auth_request con 0x7f88100b1830 auth_method 0 2023-09-30T00:49:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.398+0000 7f880ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.398+0000 7f880ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.398+0000 7f880ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.398+0000 7f880f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:34.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.398+0000 7f880f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:34.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.398+0000 7f880f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:34.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.398+0000 7f880effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:34.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.398+0000 7f880effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:34.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.398+0000 7f880effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:34.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.398+0000 7f880ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:34.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.398+0000 7f880ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:34.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.398+0000 7f880ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:34.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.402+0000 7f880ffff700 10 monclient(hunting): handle_auth_done global_id 4540 payload 824 2023-09-30T00:49:34.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.402+0000 7f880ffff700 1 -- 172.21.15.40:0/2893951725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88100a5fb0 msgr2=0x7f881013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:34.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.402+0000 7f880ffff700 1 --2- 172.21.15.40:0/2893951725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88100a5fb0 0x7f881013e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:34.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.402+0000 7f880ffff700 1 -- 172.21.15.40:0/2893951725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88100b1460 msgr2=0x7f88100a9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:34.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.402+0000 7f880ffff700 1 --2- 172.21.15.40:0/2893951725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88100b1460 0x7f88100a9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:34.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.402+0000 7f880ffff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:34.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.402+0000 7f880ffff700 1 monclient: found mon.noname-b 2023-09-30T00:49:34.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.402+0000 7f880ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:34.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.402+0000 7f880ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:49:34.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.402+0000 7f880ffff700 1 -- 172.21.15.40:0/2893951725 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8808009a20 con 0x7f88100b1830 2023-09-30T00:49:34.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.402+0000 7f880f7fe700 1 --2- 172.21.15.40:0/2893951725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88100a5fb0 0x7f881013e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:34.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.405+0000 7f880ffff700 1 --2- 172.21.15.40:0/2893951725 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88100b1830 0x7f88100aa420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f88100b1f90 tx=0x7f8808018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:34.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.405+0000 7f87f77fe700 10 client.?.objecter ms_handle_connect 0x7f88100b1830 2023-09-30T00:49:34.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.405+0000 7f87f77fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:34.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f87f77fe700 1 -- 172.21.15.40:0/2893951725 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808027470 con 0x7f88100b1830 2023-09-30T00:49:34.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f87f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:34.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f87f77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:34.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f87f77fe700 10 monclient: dump: 2023-09-30T00:49:34.410 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:34.410 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:34.410 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:34.410 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:34.410 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:34.410 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:34.410 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:34.410 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:34.410 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:34.410 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:34.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f87f77fe700 10 monclient: _finish_auth 0 2023-09-30T00:49:34.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f87f77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:34.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f88161b2700 5 monclient: authenticate success, global_id 4540 2023-09-30T00:49:34.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f88161b2700 10 monclient: _renew_subs 2023-09-30T00:49:34.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f88161b2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:34.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f88161b2700 1 -- 172.21.15.40:0/2893951725 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f881013eff0 con 0x7f88100b1830 2023-09-30T00:49:34.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f88161b2700 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:34.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f88161b2700 10 monclient: _renew_subs 2023-09-30T00:49:34.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f88161b2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:34.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f88161b2700 1 -- 172.21.15.40:0/2893951725 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88100a9420 con 0x7f88100b1830 2023-09-30T00:49:34.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.408+0000 7f88161b2700 1 librados: init done 2023-09-30T00:49:34.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.409+0000 7f88161b2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:34.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.409+0000 7f88161b2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.409+0000 7f88161b2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.409+0000 7f88161b2700 1 -- 172.21.15.40:0/2893951725 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88100a9c10 con 0x7f88100b1830 2023-09-30T00:49:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.411+0000 7f87f77fe700 1 -- 172.21.15.40:0/2893951725 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88080277e0 con 0x7f88100b1830 2023-09-30T00:49:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.411+0000 7f87f77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.411+0000 7f87f77fe700 1 -- 172.21.15.40:0/2893951725 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808027950 con 0x7f88100b1830 2023-09-30T00:49:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.411+0000 7f87f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.411+0000 7f87f77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:49:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.411+0000 7f87f77fe700 10 monclient: dump: 2023-09-30T00:49:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:34.416 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:34.416 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:34.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.411+0000 7f880dffb700 4 set_mon_vals no callback set 2023-09-30T00:49:34.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.662+0000 7f87f77fe700 1 -- 172.21.15.40:0/2893951725 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8808024070 con 0x7f88100b1830 2023-09-30T00:49:34.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.662+0000 7f87f77fe700 1 --2- 172.21.15.40:0/2893951725 >> v2:172.21.15.135:6800/116426 conn(0x7f87fc068890 0x7f87fc06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:34.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.663+0000 7f880f7fe700 1 --2- 172.21.15.40:0/2893951725 >> v2:172.21.15.135:6800/116426 conn(0x7f87fc068890 0x7f87fc06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:34.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.663+0000 7f880f7fe700 10 monclient: get_auth_request con 0x7f87fc068890 auth_method 0 2023-09-30T00:49:34.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.663+0000 7f87f77fe700 10 client.4540.objecter ms_handle_connect 0x7f87fc068890 2023-09-30T00:49:34.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.663+0000 7f880f7fe700 1 --2- 172.21.15.40:0/2893951725 >> v2:172.21.15.135:6800/116426 conn(0x7f87fc068890 0x7f87fc06ad40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8800009f60 tx=0x7f88000091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:34.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.667+0000 7f87f77fe700 1 -- 172.21.15.40:0/2893951725 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8808058540 con 0x7f88100b1830 2023-09-30T00:49:34.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.667+0000 7f87f77fe700 10 client.4540.objecter ms_dispatch 0x7f8810001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:34.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.667+0000 7f87f77fe700 3 client.4540.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:34.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.667+0000 7f87f77fe700 3 client.4540.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:34.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.667+0000 7f87f77fe700 20 client.4540.objecter dump_active .. 0 homeless 2023-09-30T00:49:34.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.676+0000 7f87f77fe700 1 -- 172.21.15.40:0/2893951725 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8808053fb0 con 0x7f88100b1830 2023-09-30T00:49:34.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.677+0000 7f87f77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:34.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.677+0000 7f87f77fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:34.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.930+0000 7f88161b2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:34.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.930+0000 7f88161b2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:34.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.930+0000 7f88161b2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:34.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.930+0000 7f88161b2700 1 -- 172.21.15.40:0/2893951725 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8810064a00 con 0x7f88100b1830 2023-09-30T00:49:34.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.936+0000 7f87f77fe700 1 -- 172.21.15.40:0/2893951725 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f88080577a0 con 0x7f88100b1830 2023-09-30T00:49:34.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.936+0000 7f87f77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:34.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.936+0000 7f87f77fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:34.936 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:34.936 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:34.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 10 librados: watch_flush enter 2023-09-30T00:49:34.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 10 librados: watch_flush exit 2023-09-30T00:49:34.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 20 client.4540.objecter shutdown clearing up homeless session... 2023-09-30T00:49:34.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 10 client.4540.objecter successfully canceled tick 2023-09-30T00:49:34.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 1 -- 172.21.15.40:0/2893951725 >> v2:172.21.15.135:6800/116426 conn(0x7f87fc068890 msgr2=0x7f87fc06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:34.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 1 --2- 172.21.15.40:0/2893951725 >> v2:172.21.15.135:6800/116426 conn(0x7f87fc068890 0x7f87fc06ad40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8800009f60 tx=0x7f88000091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 10 monclient: shutdown 2023-09-30T00:49:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 1 -- 172.21.15.40:0/2893951725 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88100b1830 msgr2=0x7f88100aa420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 1 --2- 172.21.15.40:0/2893951725 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88100b1830 0x7f88100aa420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f88100b1f90 tx=0x7f8808018000 comp rx=0 tx=0).stop 2023-09-30T00:49:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 1 -- 172.21.15.40:0/2893951725 shutdown_connections 2023-09-30T00:49:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 1 --2- 172.21.15.40:0/2893951725 >> v2:172.21.15.135:6800/116426 conn(0x7f87fc068890 0x7f87fc06ad40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 1 --2- 172.21.15.40:0/2893951725 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f88100b1830 0x7f88100aa420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 1 --2- 172.21.15.40:0/2893951725 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f88100a5fb0 0x7f881013e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 1 --2- 172.21.15.40:0/2893951725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f88100b1460 0x7f88100a9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 1 -- 172.21.15.40:0/2893951725 >> 172.21.15.40:0/2893951725 conn(0x7f88100631f0 msgr2=0x7f88100b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:34.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 1 -- 172.21.15.40:0/2893951725 shutdown_connections 2023-09-30T00:49:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 1 -- 172.21.15.40:0/2893951725 wait complete. 2023-09-30T00:49:34.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:34.938+0000 7f88161b2700 1 librados: shutdown 2023-09-30T00:49:37.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:38.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.092+0000 7f9aea442700 10 monclient: get_monmap_and_config 2023-09-30T00:49:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.092+0000 7f9aea442700 10 monclient: build_initial_monmap 2023-09-30T00:49:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.092+0000 7f9aea442700 10 monclient: monmap: 2023-09-30T00:49:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:38.093418+0000 2023-09-30T00:49:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:38.093418+0000 2023-09-30T00:49:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:38.093 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:38.094 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:38.094 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:38.094 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:38.094 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:38.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.093+0000 7f9aea442700 1 Processor -- start 2023-09-30T00:49:38.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.093+0000 7f9aea442700 1 -- start start 2023-09-30T00:49:38.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.093+0000 7f9aea442700 10 monclient: init 2023-09-30T00:49:38.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.094+0000 7f9aea442700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:38.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.094+0000 7f9aea442700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-09-30T00:49:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.094+0000 7f9aea442700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ae40b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:49:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.094+0000 7f9aea442700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ae40b1830 addr v1:172.21.15.135:6789/0 2023-09-30T00:49:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.094+0000 7f9aea442700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9ae40a5fb0 0x7f9ae413b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.094+0000 7f9aea442700 10 monclient(hunting): picked mon.noname-c con 0x7f9ae40a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:49:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.094+0000 7f9aea442700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae40a5b20 con 0x7f9ae40b1460 2023-09-30T00:49:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.094+0000 7f9aea442700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae40a71f0 con 0x7f9ae40b1830 2023-09-30T00:49:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.094+0000 7f9aea442700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.094+0000 7f9aea442700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae4141e40 con 0x7f9ae40a5fb0 2023-09-30T00:49:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.094+0000 7f9aea442700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:38.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.094+0000 7f9aea442700 10 monclient(hunting): authenticate will time out at 5439.395591s 2023-09-30T00:49:38.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.095+0000 7f9ae89df700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9ae40a5fb0 0x7f9ae413b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.098+0000 7f9ae3fff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9ae40b1460 0x7f9ae4141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36230/0 (socket says 172.21.15.40:36230) 2023-09-30T00:49:38.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.098+0000 7f9ae3fff700 1 -- 172.21.15.40:0/2909770411 learned_addr learned my addr 172.21.15.40:0/2909770411 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:49:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.102+0000 7f9ae2ffd700 1 -- 172.21.15.40:0/2909770411 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198710414 0 0) 0x7f9ae40a5b20 con 0x7f9ae40b1460 2023-09-30T00:49:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.102+0000 7f9ae2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.102+0000 7f9ae2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.102+0000 7f9ae2ffd700 10 monclient(hunting): my global_id is 4491 2023-09-30T00:49:38.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.102+0000 7f9ae2ffd700 1 -- 172.21.15.40:0/2909770411 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9acc003570 con 0x7f9ae40b1460 2023-09-30T00:49:38.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 -- 172.21.15.40:0/2909770411 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1109269913 0 0) 0x7f9acc003570 con 0x7f9ae40b1460 2023-09-30T00:49:38.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 -- 172.21.15.40:0/2909770411 >> v2:172.21.15.135:3300/0 conn(0x7f9ae40a5fb0 msgr2=0x7f9ae413b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:38.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 --2- 172.21.15.40:0/2909770411 >> v2:172.21.15.135:3300/0 conn(0x7f9ae40a5fb0 0x7f9ae413b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 -- 172.21.15.40:0/2909770411 >> v1:172.21.15.135:6789/0 conn(0x7f9ae40b1830 legacy=0x7f9ae413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 monclient: found mon.noname-a-legacy 2023-09-30T00:49:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.40:6789/0 2023-09-30T00:49:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 -- 172.21.15.40:0/2909770411 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae40a74d0 con 0x7f9ae40b1460 2023-09-30T00:49:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient: _finish_auth 0 2023-09-30T00:49:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9aea442700 5 monclient: authenticate success, global_id 4491 2023-09-30T00:49:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 -- 172.21.15.40:0/2909770411 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3033000540 0 0) 0x7f9ad4001470 con 0x7f9ae40b1460 2023-09-30T00:49:38.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T00:49:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient: dump: 2023-09-30T00:49:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:38.112 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:38.112 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] but i'm connected to v1:172.21.15.40:6789/0, reconnecting 2023-09-30T00:49:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 -- 172.21.15.40:0/2909770411 >> v1:172.21.15.40:6789/0 conn(0x7f9ae40b1460 legacy=0x7f9ae4141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:49:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 --2- 172.21.15.40:0/2909770411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9acc003b60 0x7f9acc003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient(hunting): picked mon.a con 0x7f9acc003b60 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 --2- 172.21.15.40:0/2909770411 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9acc004810 0x7f9acc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient(hunting): picked mon.b con 0x7f9acc004810 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 --2- 172.21.15.40:0/2909770411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9acc005410 0x7f9acc00bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient(hunting): picked mon.c con 0x7f9acc005410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:38.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 -- 172.21.15.40:0/2909770411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae4141e40 con 0x7f9acc003b60 2023-09-30T00:49:38.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 -- 172.21.15.40:0/2909770411 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc00c3e0 con 0x7f9acc004810 2023-09-30T00:49:38.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 1 -- 172.21.15.40:0/2909770411 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9acc00c5a0 con 0x7f9acc005410 2023-09-30T00:49:38.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:38.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9ae2ffd700 10 monclient(hunting): _finish_auth 0 2023-09-30T00:49:38.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.109+0000 7f9aea442700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:38.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.115+0000 7f9ae3fff700 1 --2- 172.21.15.40:0/2909770411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9acc003b60 0x7f9acc003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.115+0000 7f9ae3fff700 10 monclient(hunting): get_auth_request con 0x7f9acc003b60 auth_method 0 2023-09-30T00:49:38.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.115+0000 7f9ae3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.115+0000 7f9ae3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.115+0000 7f9ae3fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:49:38.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.116+0000 7f9ae3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:38.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.116+0000 7f9ae3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:38.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.116+0000 7f9ae3fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:49:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.119+0000 7f9ae3fff700 10 monclient(hunting): handle_auth_done global_id 4491 payload 293 2023-09-30T00:49:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.119+0000 7f9ae3fff700 1 -- 172.21.15.40:0/2909770411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9acc005410 msgr2=0x7f9acc00bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.119+0000 7f9ae3fff700 1 --2- 172.21.15.40:0/2909770411 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9acc005410 0x7f9acc00bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.119+0000 7f9ae3fff700 1 -- 172.21.15.40:0/2909770411 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9acc004810 msgr2=0x7f9acc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.119+0000 7f9ae3fff700 1 --2- 172.21.15.40:0/2909770411 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9acc004810 0x7f9acc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.119+0000 7f9ae3fff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.119+0000 7f9ae3fff700 1 monclient: found mon.a 2023-09-30T00:49:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.119+0000 7f9ae3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.119+0000 7f9ae3fff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:49:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.119+0000 7f9ae3fff700 1 -- 172.21.15.40:0/2909770411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9acc003570 con 0x7f9acc003b60 2023-09-30T00:49:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.119+0000 7f9ae89df700 1 -- 172.21.15.40:0/2909770411 reap_dead start 2023-09-30T00:49:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.119+0000 7f9ae3fff700 10 monclient: _finish_auth 0 2023-09-30T00:49:38.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.119+0000 7f9ae3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:38.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.122+0000 7f9ae3fff700 1 --2- 172.21.15.40:0/2909770411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9acc003b60 0x7f9acc003f50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400c2d0 tx=0x7f9ad4001da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a94302d837e9929 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9ae2ffd700 1 -- 172.21.15.40:0/2909770411 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad400cba0 con 0x7f9acc003b60 2023-09-30T00:49:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9ae2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9ae2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9ae2ffd700 10 monclient: dump: 2023-09-30T00:49:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:38.128 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:38.129 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9ae2ffd700 1 -- 172.21.15.40:0/2909770411 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad400cd20 con 0x7f9acc003b60 2023-09-30T00:49:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9aea442700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9ae2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9aea442700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9aea442700 4 set_mon_vals no callback set 2023-09-30T00:49:38.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9ae2ffd700 1 -- 172.21.15.40:0/2909770411 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4005b70 con 0x7f9acc003b60 2023-09-30T00:49:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9ae2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9ae2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9ae2ffd700 10 monclient: dump: 2023-09-30T00:49:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:38.130 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:38.130 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:38.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9aea442700 10 monclient: shutdown 2023-09-30T00:49:38.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9aea442700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:38.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9aea442700 1 -- 172.21.15.40:0/2909770411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9acc003b60 msgr2=0x7f9acc003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9aea442700 1 --2- 172.21.15.40:0/2909770411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9acc003b60 0x7f9acc003f50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9ad400c2d0 tx=0x7f9ad4001da0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9aea442700 1 -- 172.21.15.40:0/2909770411 shutdown_connections 2023-09-30T00:49:38.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9aea442700 1 --2- 172.21.15.40:0/2909770411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9acc003b60 0x7f9acc003f50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.127+0000 7f9aea442700 1 -- 172.21.15.40:0/2909770411 >> 172.21.15.40:0/2909770411 conn(0x7f9ae40631f0 msgr2=0x7f9ae40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:38.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.129+0000 7f9aea442700 1 -- 172.21.15.40:0/2909770411 shutdown_connections 2023-09-30T00:49:38.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.129+0000 7f9aea442700 1 -- 172.21.15.40:0/2909770411 wait complete. 2023-09-30T00:49:38.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.129+0000 7f9aea442700 10 monclient: build_initial_monmap 2023-09-30T00:49:38.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.129+0000 7f9aea442700 10 monclient: monmap: 2023-09-30T00:49:38.133 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:38.133 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:38.133 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:38.134 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:38.134 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:38.134 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:38.134 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:38.134 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:38.134 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:38.134 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:38.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.130+0000 7f9aea442700 1 librados: starting msgr at 2023-09-30T00:49:38.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.130+0000 7f9aea442700 1 librados: starting objecter 2023-09-30T00:49:38.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.130+0000 7f9aea442700 1 Processor -- start 2023-09-30T00:49:38.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.130+0000 7f9aea442700 1 -- start start 2023-09-30T00:49:38.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.130+0000 7f9aea442700 1 librados: setting wanted keys 2023-09-30T00:49:38.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.130+0000 7f9aea442700 1 librados: calling monclient init 2023-09-30T00:49:38.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.130+0000 7f9aea442700 10 monclient: init 2023-09-30T00:49:38.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:38.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:49:38.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9acc003b60 0x7f9ae40a9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 10 monclient(hunting): picked mon.noname-a con 0x7f9acc003b60 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:38.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ae40a8b40 0x7f9ae40a9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 10 monclient(hunting): picked mon.noname-b con 0x7f9ae40a8b40 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:38.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ae40aaf70 0x7f9ae40ab3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 10 monclient(hunting): picked mon.noname-c con 0x7f9ae40aaf70 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:38.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40aa870 con 0x7f9acc003b60 2023-09-30T00:49:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40aab50 con 0x7f9ae40a8b40 2023-09-30T00:49:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae40a8620 con 0x7f9ae40aaf70 2023-09-30T00:49:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9aea442700 10 monclient(hunting): authenticate will time out at 5439.432304s 2023-09-30T00:49:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.131+0000 7f9ae3fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9acc003b60 0x7f9ae40a9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.132+0000 7f9ae3fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9acc003b60 0x7f9ae40a9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52854/0 (socket says 172.21.15.40:52854) 2023-09-30T00:49:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.132+0000 7f9ae3fff700 1 -- 172.21.15.40:0/1873436127 learned_addr learned my addr 172.21.15.40:0/1873436127 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.132+0000 7f9ae3fff700 10 monclient(hunting): get_auth_request con 0x7f9acc003b60 auth_method 0 2023-09-30T00:49:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.132+0000 7f9ae3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.132+0000 7f9ae3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.132+0000 7f9ae3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.132+0000 7f9ae3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:38.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.132+0000 7f9ae3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:38.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.132+0000 7f9ae3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:38.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.136+0000 7f9ae3fff700 10 monclient(hunting): handle_auth_done global_id 4494 payload 824 2023-09-30T00:49:38.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.136+0000 7f9ae3fff700 1 -- 172.21.15.40:0/1873436127 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ae40aaf70 msgr2=0x7f9ae40ab3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.136+0000 7f9ae3fff700 1 --2- 172.21.15.40:0/1873436127 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ae40aaf70 0x7f9ae40ab3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.136+0000 7f9ae3fff700 1 -- 172.21.15.40:0/1873436127 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ae40a8b40 msgr2=0x7f9ae40a9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.136+0000 7f9ae3fff700 1 --2- 172.21.15.40:0/1873436127 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ae40a8b40 0x7f9ae40a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.136+0000 7f9ae3fff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:38.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.136+0000 7f9ae3fff700 1 monclient: found mon.noname-a 2023-09-30T00:49:38.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.136+0000 7f9ae3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:38.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.136+0000 7f9ae3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:49:38.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.136+0000 7f9ae3fff700 1 -- 172.21.15.40:0/1873436127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae406e210 con 0x7f9acc003b60 2023-09-30T00:49:38.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.139+0000 7f9ae3fff700 1 --2- 172.21.15.40:0/1873436127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9acc003b60 0x7f9ae40a9010 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9acc00c3b0 tx=0x7f9ad4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:38.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.139+0000 7f9acbfff700 10 client.?.objecter ms_handle_connect 0x7f9acc003b60 2023-09-30T00:49:38.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.139+0000 7f9acbfff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:38.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9acbfff700 1 -- 172.21.15.40:0/1873436127 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4022070 con 0x7f9acc003b60 2023-09-30T00:49:38.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9acbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9acbfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:38.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9acbfff700 10 monclient: dump: 2023-09-30T00:49:38.145 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:38.145 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:38.145 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:38.145 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:38.145 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.145 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:38.146 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:38.146 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:38.146 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:38.146 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:38.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9acbfff700 10 monclient: _finish_auth 0 2023-09-30T00:49:38.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9acbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:38.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9acbfff700 1 -- 172.21.15.40:0/1873436127 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad400cba0 con 0x7f9acc003b60 2023-09-30T00:49:38.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9aea442700 5 monclient: authenticate success, global_id 4494 2023-09-30T00:49:38.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9aea442700 10 monclient: _renew_subs 2023-09-30T00:49:38.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9aea442700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:49:38.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9aea442700 1 -- 172.21.15.40:0/1873436127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae406e400 con 0x7f9acc003b60 2023-09-30T00:49:38.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9acbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9acbfff700 1 -- 172.21.15.40:0/1873436127 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad4007530 con 0x7f9acc003b60 2023-09-30T00:49:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9acbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9acbfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9acbfff700 10 monclient: dump: 2023-09-30T00:49:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:38.147 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:38.148 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:38.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9ae27fc700 4 set_mon_vals no callback set 2023-09-30T00:49:38.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9aea442700 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:38.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9aea442700 10 monclient: _renew_subs 2023-09-30T00:49:38.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9aea442700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:49:38.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9aea442700 1 -- 172.21.15.40:0/1873436127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae406cef0 con 0x7f9acc003b60 2023-09-30T00:49:38.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.143+0000 7f9aea442700 1 librados: init done 2023-09-30T00:49:38.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.145+0000 7f9aea442700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:38.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.145+0000 7f9aea442700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:38.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.145+0000 7f9aea442700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:49:38.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.145+0000 7f9aea442700 1 -- 172.21.15.40:0/1873436127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae4064a00 con 0x7f9acc003b60 2023-09-30T00:49:38.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.455+0000 7f9acbfff700 1 -- 172.21.15.40:0/1873436127 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9ad401e430 con 0x7f9acc003b60 2023-09-30T00:49:38.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.455+0000 7f9acbfff700 1 --2- 172.21.15.40:0/1873436127 >> v2:172.21.15.135:6800/116426 conn(0x7f9ad00689e0 0x7f9ad006ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.455+0000 7f9acbfff700 1 -- 172.21.15.40:0/1873436127 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9ad4084e60 con 0x7f9acc003b60 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.455+0000 7f9acbfff700 10 client.4494.objecter ms_dispatch 0x7f9ae4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.455+0000 7f9acbfff700 3 client.4494.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.455+0000 7f9acbfff700 3 client.4494.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.455+0000 7f9acbfff700 20 client.4494.objecter dump_active .. 0 homeless 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.455+0000 7f9acbfff700 1 -- 172.21.15.40:0/1873436127 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9ad40852f0 con 0x7f9acc003b60 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.455+0000 7f9acbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.455+0000 7f9acbfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.456+0000 7f9ae37fe700 1 --2- 172.21.15.40:0/1873436127 >> v2:172.21.15.135:6800/116426 conn(0x7f9ad00689e0 0x7f9ad006ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.456+0000 7f9ae37fe700 10 monclient: get_auth_request con 0x7f9ad00689e0 auth_method 0 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.456+0000 7f9acbfff700 10 client.4494.objecter ms_handle_connect 0x7f9ad00689e0 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.456+0000 7f9ae37fe700 1 --2- 172.21.15.40:0/1873436127 >> v2:172.21.15.135:6800/116426 conn(0x7f9ad00689e0 0x7f9ad006ae90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40ae680 tx=0x7f9ad8006c60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:38.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.724+0000 7f9aea442700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:38.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.724+0000 7f9aea442700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:38.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.724+0000 7f9aea442700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:49:38.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.724+0000 7f9aea442700 1 -- 172.21.15.40:0/1873436127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9ae40a63b0 con 0x7f9acc003b60 2023-09-30T00:49:38.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.730+0000 7f9acbfff700 1 -- 172.21.15.40:0/1873436127 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9ad4052780 con 0x7f9acc003b60 2023-09-30T00:49:38.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.730+0000 7f9acbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:38.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.730+0000 7f9acbfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:38.731 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:38.731 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:38.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 10 librados: watch_flush enter 2023-09-30T00:49:38.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 10 librados: watch_flush exit 2023-09-30T00:49:38.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 20 client.4494.objecter shutdown clearing up homeless session... 2023-09-30T00:49:38.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 10 client.4494.objecter successfully canceled tick 2023-09-30T00:49:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 1 -- 172.21.15.40:0/1873436127 >> v2:172.21.15.135:6800/116426 conn(0x7f9ad00689e0 msgr2=0x7f9ad006ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 1 --2- 172.21.15.40:0/1873436127 >> v2:172.21.15.135:6800/116426 conn(0x7f9ad00689e0 0x7f9ad006ae90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40ae680 tx=0x7f9ad8006c60 comp rx=0 tx=0).stop 2023-09-30T00:49:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 10 monclient: shutdown 2023-09-30T00:49:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 1 -- 172.21.15.40:0/1873436127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9acc003b60 msgr2=0x7f9ae40a9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 1 --2- 172.21.15.40:0/1873436127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9acc003b60 0x7f9ae40a9010 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9acc00c3b0 tx=0x7f9ad4015000 comp rx=0 tx=0).stop 2023-09-30T00:49:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 1 -- 172.21.15.40:0/1873436127 shutdown_connections 2023-09-30T00:49:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 1 --2- 172.21.15.40:0/1873436127 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ae40aaf70 0x7f9ae40ab3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 1 --2- 172.21.15.40:0/1873436127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9acc003b60 0x7f9ae40a9010 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 1 --2- 172.21.15.40:0/1873436127 >> v2:172.21.15.135:6800/116426 conn(0x7f9ad00689e0 0x7f9ad006ae90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 1 --2- 172.21.15.40:0/1873436127 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ae40a8b40 0x7f9ae40a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 1 -- 172.21.15.40:0/1873436127 >> 172.21.15.40:0/1873436127 conn(0x7f9ae40631f0 msgr2=0x7f9ae4141730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:38.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 1 -- 172.21.15.40:0/1873436127 shutdown_connections 2023-09-30T00:49:38.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 1 -- 172.21.15.40:0/1873436127 wait complete. 2023-09-30T00:49:38.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:38.733+0000 7f9aea442700 1 librados: shutdown 2023-09-30T00:49:41.749 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:41.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.888+0000 7f8d8a5b7700 10 monclient: get_monmap_and_config 2023-09-30T00:49:41.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.888+0000 7f8d8a5b7700 10 monclient: build_initial_monmap 2023-09-30T00:49:41.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.888+0000 7f8d8a5b7700 10 monclient: monmap: 2023-09-30T00:49:41.889 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:41.889 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:41.890 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:41.889672+0000 2023-09-30T00:49:41.890 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:41.889672+0000 2023-09-30T00:49:41.890 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:41.890 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:41.890 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:41.890 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:41.890 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:41.890 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:41.890 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:41.891 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:41.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.889+0000 7f8d8a5b7700 1 Processor -- start 2023-09-30T00:49:41.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.889+0000 7f8d8a5b7700 1 -- start start 2023-09-30T00:49:41.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.889+0000 7f8d8a5b7700 10 monclient: init 2023-09-30T00:49:41.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:41.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-09-30T00:49:41.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d840b1460 0x7f8d840b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 10 monclient(hunting): picked mon.noname-b con 0x7f8d840b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:41.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8d840a5fb0 0x7f8d84141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 10 monclient(hunting): picked mon.noname-c con 0x7f8d840a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:49:41.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8d840af410 0x7f8d840af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 10 monclient(hunting): picked mon.noname-a con 0x7f8d840af410 addr v2:172.21.15.40:3300/0 2023-09-30T00:49:41.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d840a71f0 con 0x7f8d840af410 2023-09-30T00:49:41.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d840a74d0 con 0x7f8d840a5fb0 2023-09-30T00:49:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d840a63c0 con 0x7f8d840b1460 2023-09-30T00:49:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.890+0000 7f8d8a5b7700 10 monclient(hunting): authenticate will time out at 5443.191713s 2023-09-30T00:49:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.891+0000 7f8d83fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d840b1460 0x7f8d840b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.891+0000 7f8d837fe700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8d840a5fb0 0x7f8d84141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d837fe700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8d840a5fb0 0x7f8d84141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:57836/0 (socket says 172.21.15.40:57836) 2023-09-30T00:49:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d83fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d840b1460 0x7f8d840b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:51378/0 (socket says 172.21.15.40:51378) 2023-09-30T00:49:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d837fe700 1 -- 172.21.15.40:0/5053612 learned_addr learned my addr 172.21.15.40:0/5053612 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:41.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d837fe700 10 monclient(hunting): get_auth_request con 0x7f8d840a5fb0 auth_method 0 2023-09-30T00:49:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d837fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d837fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d83fff700 10 monclient(hunting): get_auth_request con 0x7f8d840b1460 auth_method 0 2023-09-30T00:49:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d83fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d83fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d83fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d83fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d83fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d83fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.892+0000 7f8d837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.895+0000 7f8d83fff700 10 monclient(hunting): handle_auth_done global_id 4436 payload 274 2023-09-30T00:49:41.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.895+0000 7f8d83fff700 1 -- 172.21.15.40:0/5053612 >> v2:172.21.15.135:3300/0 conn(0x7f8d840a5fb0 msgr2=0x7f8d84141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:41.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.895+0000 7f8d83fff700 1 --2- 172.21.15.40:0/5053612 >> v2:172.21.15.135:3300/0 conn(0x7f8d840a5fb0 0x7f8d84141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.895+0000 7f8d83fff700 1 -- 172.21.15.40:0/5053612 >> v2:172.21.15.40:3300/0 conn(0x7f8d840af410 msgr2=0x7f8d840af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:41.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.895+0000 7f8d83fff700 1 --2- 172.21.15.40:0/5053612 >> v2:172.21.15.40:3300/0 conn(0x7f8d840af410 0x7f8d840af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.895+0000 7f8d83fff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:41.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.895+0000 7f8d83fff700 1 monclient: found mon.noname-b 2023-09-30T00:49:41.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.895+0000 7f8d83fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:41.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.895+0000 7f8d83fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:49:41.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.895+0000 7f8d83fff700 1 -- 172.21.15.40:0/5053612 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d840a56d0 con 0x7f8d840b1460 2023-09-30T00:49:41.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.897+0000 7f8d83fff700 1 --2- 172.21.15.40:0/5053612 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d840b1460 0x7f8d840b1830 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8d8406f560 tx=0x7f8d740091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c47549b34ea140bf server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d82ffd700 1 -- 172.21.15.40:0/5053612 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d74011630 con 0x7f8d840b1460 2023-09-30T00:49:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d82ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d82ffd700 10 monclient: dump: 2023-09-30T00:49:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:41.903 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:41.904 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d82ffd700 10 monclient: _finish_auth 0 2023-09-30T00:49:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d82ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d82ffd700 1 -- 172.21.15.40:0/5053612 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d74011790 con 0x7f8d840b1460 2023-09-30T00:49:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d82ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d82ffd700 1 -- 172.21.15.40:0/5053612 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d74011b30 con 0x7f8d840b1460 2023-09-30T00:49:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d8a5b7700 5 monclient: authenticate success, global_id 4436 2023-09-30T00:49:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d8a5b7700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:41.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d8a5b7700 4 set_mon_vals no callback set 2023-09-30T00:49:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d82ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.902+0000 7f8d82ffd700 10 monclient: dump: 2023-09-30T00:49:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:41.905 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:41.906 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.903+0000 7f8d8a5b7700 10 monclient: shutdown 2023-09-30T00:49:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.903+0000 7f8d8a5b7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.903+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/5053612 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d840b1460 msgr2=0x7f8d840b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.903+0000 7f8d8a5b7700 1 --2- 172.21.15.40:0/5053612 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d840b1460 0x7f8d840b1830 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8d8406f560 tx=0x7f8d740091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.903+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/5053612 shutdown_connections 2023-09-30T00:49:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.903+0000 7f8d8a5b7700 1 --2- 172.21.15.40:0/5053612 >> v2:172.21.15.40:3300/0 conn(0x7f8d840af410 0x7f8d840af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.903+0000 7f8d8a5b7700 1 --2- 172.21.15.40:0/5053612 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d840b1460 0x7f8d840b1830 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.903+0000 7f8d8a5b7700 1 --2- 172.21.15.40:0/5053612 >> v2:172.21.15.135:3300/0 conn(0x7f8d840a5fb0 0x7f8d84141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.903+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/5053612 >> 172.21.15.40:0/5053612 conn(0x7f8d840631f0 msgr2=0x7f8d840635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.903+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/5053612 shutdown_connections 2023-09-30T00:49:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.903+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/5053612 wait complete. 2023-09-30T00:49:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.904+0000 7f8d8a5b7700 10 monclient: build_initial_monmap 2023-09-30T00:49:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.904+0000 7f8d8a5b7700 10 monclient: monmap: 2023-09-30T00:49:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:41.910 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:41.911 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:41.911 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:41.911 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:41.911 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:41.911 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:41.911 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:41.911 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:41.911 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:41.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.904+0000 7f8d8a5b7700 1 librados: starting msgr at 2023-09-30T00:49:41.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.904+0000 7f8d8a5b7700 1 librados: starting objecter 2023-09-30T00:49:41.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.904+0000 7f8d8a5b7700 1 Processor -- start 2023-09-30T00:49:41.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 1 -- start start 2023-09-30T00:49:41.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 1 librados: setting wanted keys 2023-09-30T00:49:41.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 1 librados: calling monclient init 2023-09-30T00:49:41.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 10 monclient: init 2023-09-30T00:49:41.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:41.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:49:41.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d840a5fb0 0x7f8d8413e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 10 monclient(hunting): picked mon.noname-a con 0x7f8d840a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:41.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d840af410 0x7f8d840a7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 10 monclient(hunting): picked mon.noname-c con 0x7f8d840af410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:41.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d840b1460 0x7f8d840a83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 10 monclient(hunting): picked mon.noname-b con 0x7f8d840b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:41.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d840a8a80 con 0x7f8d840a5fb0 2023-09-30T00:49:41.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d840a9200 con 0x7f8d840b1460 2023-09-30T00:49:41.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d840a93c0 con 0x7f8d840af410 2023-09-30T00:49:41.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:41.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.905+0000 7f8d8a5b7700 10 monclient(hunting): authenticate will time out at 5443.206748s 2023-09-30T00:49:41.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.906+0000 7f8d88b54700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d840b1460 0x7f8d840a83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.906+0000 7f8d88b54700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d840b1460 0x7f8d840a83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:57852/0 (socket says 172.21.15.40:57852) 2023-09-30T00:49:41.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.906+0000 7f8d88b54700 1 -- 172.21.15.40:0/234693886 learned_addr learned my addr 172.21.15.40:0/234693886 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:41.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.906+0000 7f8d88b54700 10 monclient(hunting): get_auth_request con 0x7f8d840b1460 auth_method 0 2023-09-30T00:49:41.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.906+0000 7f8d88b54700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.906+0000 7f8d88b54700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.906+0000 7f8d88b54700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.907+0000 7f8d88b54700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.907+0000 7f8d88b54700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.907+0000 7f8d88b54700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.907+0000 7f8d837fe700 1 --2- 172.21.15.40:0/234693886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d840af410 0x7f8d840a7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.908+0000 7f8d837fe700 10 monclient(hunting): get_auth_request con 0x7f8d840af410 auth_method 0 2023-09-30T00:49:41.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.908+0000 7f8d837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.908+0000 7f8d837fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.908+0000 7f8d837fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.908+0000 7f8d837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.908+0000 7f8d837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.908+0000 7f8d837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.910+0000 7f8d88b54700 10 monclient(hunting): handle_auth_done global_id 4546 payload 824 2023-09-30T00:49:41.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.910+0000 7f8d88b54700 1 -- 172.21.15.40:0/234693886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d840af410 msgr2=0x7f8d840a7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.910+0000 7f8d88b54700 1 --2- 172.21.15.40:0/234693886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d840af410 0x7f8d840a7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.910+0000 7f8d88b54700 1 -- 172.21.15.40:0/234693886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d840a5fb0 msgr2=0x7f8d8413e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.910+0000 7f8d88b54700 1 --2- 172.21.15.40:0/234693886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d840a5fb0 0x7f8d8413e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.910+0000 7f8d88b54700 10 monclient: _finish_hunting 0 2023-09-30T00:49:41.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.910+0000 7f8d88b54700 1 monclient: found mon.noname-b 2023-09-30T00:49:41.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.910+0000 7f8d88b54700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:41.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.911+0000 7f8d88b54700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:49:41.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.911+0000 7f8d88b54700 1 -- 172.21.15.40:0/234693886 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d74009420 con 0x7f8d840b1460 2023-09-30T00:49:41.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.913+0000 7f8d88b54700 1 --2- 172.21.15.40:0/234693886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d840b1460 0x7f8d840a83e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c00bc30 tx=0x7f8d7c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:41.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.913+0000 7f8d6bfff700 10 client.?.objecter ms_handle_connect 0x7f8d840b1460 2023-09-30T00:49:41.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.913+0000 7f8d6bfff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:41.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d6bfff700 1 -- 172.21.15.40:0/234693886 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c0092e0 con 0x7f8d840b1460 2023-09-30T00:49:41.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d6bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d6bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:41.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d6bfff700 10 monclient: dump: 2023-09-30T00:49:41.918 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:41.918 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:41.918 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:41.918 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:41.918 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.918 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:41.918 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:41.918 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:41.918 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:41.919 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:41.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d6bfff700 10 monclient: _finish_auth 0 2023-09-30T00:49:41.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d6bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:41.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d8a5b7700 5 monclient: authenticate success, global_id 4546 2023-09-30T00:49:41.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d8a5b7700 10 monclient: _renew_subs 2023-09-30T00:49:41.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d8a5b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:41.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/234693886 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d840a9640 con 0x7f8d840b1460 2023-09-30T00:49:41.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d8a5b7700 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:41.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d8a5b7700 10 monclient: _renew_subs 2023-09-30T00:49:41.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d8a5b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:41.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/234693886 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d840a9bb0 con 0x7f8d840b1460 2023-09-30T00:49:41.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.916+0000 7f8d8a5b7700 1 librados: init done 2023-09-30T00:49:41.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.918+0000 7f8d8a5b7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:41.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.918+0000 7f8d8a5b7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:41.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.918+0000 7f8d8a5b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:41.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.918+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/234693886 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d840ab310 con 0x7f8d840b1460 2023-09-30T00:49:41.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.919+0000 7f8d6bfff700 1 -- 172.21.15.40:0/234693886 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7c00c900 con 0x7f8d840b1460 2023-09-30T00:49:41.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.919+0000 7f8d6bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:41.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.919+0000 7f8d6bfff700 1 -- 172.21.15.40:0/234693886 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c00ca90 con 0x7f8d840b1460 2023-09-30T00:49:41.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.919+0000 7f8d6bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.919+0000 7f8d6bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:49:41.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.919+0000 7f8d6bfff700 10 monclient: dump: 2023-09-30T00:49:41.921 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:41.922 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:41.922 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:41.922 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:41.922 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.922 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:41.922 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:41.922 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:41.922 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:41.922 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:41.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:41.919+0000 7f8d827fc700 4 set_mon_vals no callback set 2023-09-30T00:49:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.162+0000 7f8d6bfff700 1 -- 172.21.15.40:0/234693886 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8d7c012070 con 0x7f8d840b1460 2023-09-30T00:49:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.162+0000 7f8d6bfff700 1 --2- 172.21.15.40:0/234693886 >> v2:172.21.15.135:6800/116426 conn(0x7f8d70068890 0x7f8d7006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.163+0000 7f8d83fff700 1 --2- 172.21.15.40:0/234693886 >> v2:172.21.15.135:6800/116426 conn(0x7f8d70068890 0x7f8d7006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.163+0000 7f8d83fff700 10 monclient: get_auth_request con 0x7f8d70068890 auth_method 0 2023-09-30T00:49:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.163+0000 7f8d6bfff700 10 client.4546.objecter ms_handle_connect 0x7f8d70068890 2023-09-30T00:49:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.163+0000 7f8d83fff700 1 --2- 172.21.15.40:0/234693886 >> v2:172.21.15.135:6800/116426 conn(0x7f8d70068890 0x7f8d7006ad40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8d8406f560 tx=0x7f8d74017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:42.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.203+0000 7f8d6bfff700 1 -- 172.21.15.40:0/234693886 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8d7c01c2f0 con 0x7f8d840b1460 2023-09-30T00:49:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.203+0000 7f8d6bfff700 10 client.4546.objecter ms_dispatch 0x7f8d84001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.203+0000 7f8d6bfff700 3 client.4546.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.203+0000 7f8d6bfff700 3 client.4546.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:42.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.204+0000 7f8d6bfff700 20 client.4546.objecter dump_active .. 0 homeless 2023-09-30T00:49:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.213+0000 7f8d6bfff700 1 -- 172.21.15.40:0/234693886 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d7c04e5a0 con 0x7f8d840b1460 2023-09-30T00:49:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.213+0000 7f8d6bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:42.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.213+0000 7f8d6bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.469+0000 7f8d8a5b7700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.470+0000 7f8d8a5b7700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.470+0000 7f8d8a5b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.470+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/234693886 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8d840a8f00 con 0x7f8d840b1460 2023-09-30T00:49:42.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.476+0000 7f8d6bfff700 1 -- 172.21.15.40:0/234693886 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8d7c054da0 con 0x7f8d840b1460 2023-09-30T00:49:42.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.476+0000 7f8d6bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:42.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.476+0000 7f8d6bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:42.477 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:42.477 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 10 librados: watch_flush enter 2023-09-30T00:49:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 10 librados: watch_flush exit 2023-09-30T00:49:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 20 client.4546.objecter shutdown clearing up homeless session... 2023-09-30T00:49:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 10 client.4546.objecter successfully canceled tick 2023-09-30T00:49:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/234693886 >> v2:172.21.15.135:6800/116426 conn(0x7f8d70068890 msgr2=0x7f8d7006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 1 --2- 172.21.15.40:0/234693886 >> v2:172.21.15.135:6800/116426 conn(0x7f8d70068890 0x7f8d7006ad40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8d8406f560 tx=0x7f8d74017000 comp rx=0 tx=0).stop 2023-09-30T00:49:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 10 monclient: shutdown 2023-09-30T00:49:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/234693886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d840b1460 msgr2=0x7f8d840a83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 1 --2- 172.21.15.40:0/234693886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d840b1460 0x7f8d840a83e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c00bc30 tx=0x7f8d7c009840 comp rx=0 tx=0).stop 2023-09-30T00:49:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/234693886 shutdown_connections 2023-09-30T00:49:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 1 --2- 172.21.15.40:0/234693886 >> v2:172.21.15.135:6800/116426 conn(0x7f8d70068890 0x7f8d7006ad40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 1 --2- 172.21.15.40:0/234693886 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8d840b1460 0x7f8d840a83e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 1 --2- 172.21.15.40:0/234693886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d840a5fb0 0x7f8d8413e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 1 --2- 172.21.15.40:0/234693886 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8d840af410 0x7f8d840a7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/234693886 >> 172.21.15.40:0/234693886 conn(0x7f8d840631f0 msgr2=0x7f8d840b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/234693886 shutdown_connections 2023-09-30T00:49:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 1 -- 172.21.15.40:0/234693886 wait complete. 2023-09-30T00:49:42.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:42.478+0000 7f8d8a5b7700 1 librados: shutdown 2023-09-30T00:49:45.493 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.633+0000 7fbbe2757700 10 monclient: get_monmap_and_config 2023-09-30T00:49:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.633+0000 7fbbe2757700 10 monclient: build_initial_monmap 2023-09-30T00:49:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.633+0000 7fbbe2757700 10 monclient: monmap: 2023-09-30T00:49:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:45.634278+0000 2023-09-30T00:49:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:45.634278+0000 2023-09-30T00:49:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:45.635 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 1 Processor -- start 2023-09-30T00:49:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 1 -- start start 2023-09-30T00:49:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 10 monclient: init 2023-09-30T00:49:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-09-30T00:49:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbbdc0b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:49:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbbdc0b1830 0x7fbbdc0a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 10 monclient(hunting): picked mon.noname-c con 0x7fbbdc0b1830 addr v2:172.21.15.135:3300/0 2023-09-30T00:49:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbbdc13ac20 0x7fbbdc13b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 10 monclient(hunting): picked mon.noname-b con 0x7fbbdc13ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbdc0a5b20 con 0x7fbbdc0b1460 2023-09-30T00:49:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbdc141e40 con 0x7fbbdc0b1830 2023-09-30T00:49:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbdc0a64f0 con 0x7fbbdc13ac20 2023-09-30T00:49:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.634+0000 7fbbe2757700 10 monclient(hunting): authenticate will time out at 5446.935773s 2023-09-30T00:49:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.635+0000 7fbbdbfff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbbdc0b1460 0x7fbbdc141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37934/0 (socket says 172.21.15.40:37934) 2023-09-30T00:49:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.635+0000 7fbbe0cf4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbbdc13ac20 0x7fbbdc13b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.635+0000 7fbbdbfff700 1 -- 172.21.15.40:0/1363300981 learned_addr learned my addr 172.21.15.40:0/1363300981 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:49:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.635+0000 7fbbdb7fe700 1 --2- 172.21.15.40:0/1363300981 >> v2:172.21.15.135:3300/0 conn(0x7fbbdc0b1830 0x7fbbdc0a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.636+0000 7fbbe0cf4700 10 monclient(hunting): get_auth_request con 0x7fbbdc13ac20 auth_method 0 2023-09-30T00:49:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.636+0000 7fbbe0cf4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.636+0000 7fbbe0cf4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.636+0000 7fbbe0cf4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.636+0000 7fbbdb7fe700 10 monclient(hunting): get_auth_request con 0x7fbbdc0b1830 auth_method 0 2023-09-30T00:49:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.636+0000 7fbbdb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.636+0000 7fbbdb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.636+0000 7fbbdb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.636+0000 7fbbe0cf4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.636+0000 7fbbe0cf4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.636+0000 7fbbe0cf4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.636+0000 7fbbdb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.636+0000 7fbbdb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.636+0000 7fbbdb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.639+0000 7fbbdaffd700 1 -- 172.21.15.40:0/1363300981 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872101257 0 0) 0x7fbbdc0a5b20 con 0x7fbbdc0b1460 2023-09-30T00:49:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.639+0000 7fbbdaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.639+0000 7fbbdaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.639+0000 7fbbdaffd700 10 monclient(hunting): my global_id is 4497 2023-09-30T00:49:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.639+0000 7fbbdaffd700 1 -- 172.21.15.40:0/1363300981 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbc4003570 con 0x7fbbdc0b1460 2023-09-30T00:49:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.639+0000 7fbbdb7fe700 10 monclient(hunting): handle_auth_done global_id 4549 payload 274 2023-09-30T00:49:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.639+0000 7fbbdb7fe700 1 -- 172.21.15.40:0/1363300981 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbbdc13ac20 msgr2=0x7fbbdc13b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.639+0000 7fbbdb7fe700 1 --2- 172.21.15.40:0/1363300981 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbbdc13ac20 0x7fbbdc13b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.639+0000 7fbbdb7fe700 1 -- 172.21.15.40:0/1363300981 >> v1:172.21.15.40:6789/0 conn(0x7fbbdc0b1460 legacy=0x7fbbdc141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.639+0000 7fbbdb7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:49:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.639+0000 7fbbdb7fe700 1 monclient: found mon.noname-c 2023-09-30T00:49:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.639+0000 7fbbdb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.639+0000 7fbbdb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:49:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.639+0000 7fbbdb7fe700 1 -- 172.21.15.40:0/1363300981 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbdc0a71f0 con 0x7fbbdc0b1830 2023-09-30T00:49:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.641+0000 7fbbdb7fe700 1 --2- 172.21.15.40:0/1363300981 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbdc0b1830 0x7fbbdc0a5fb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0009910 tx=0x7fbbd00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb8e31a59a856ba0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:45.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbdaffd700 1 -- 172.21.15.40:0/1363300981 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0011630 con 0x7fbbdc0b1830 2023-09-30T00:49:45.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbdaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:45.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbdaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:45.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbdaffd700 10 monclient: dump: 2023-09-30T00:49:45.647 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:45.647 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:45.647 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:45.648 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbdaffd700 10 monclient: _finish_auth 0 2023-09-30T00:49:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbdaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbdaffd700 1 -- 172.21.15.40:0/1363300981 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd0011790 con 0x7fbbdc0b1830 2023-09-30T00:49:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbdaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:45.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbdaffd700 1 -- 172.21.15.40:0/1363300981 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0011b30 con 0x7fbbdc0b1830 2023-09-30T00:49:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbe2757700 5 monclient: authenticate success, global_id 4549 2023-09-30T00:49:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbe2757700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbe2757700 4 set_mon_vals no callback set 2023-09-30T00:49:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbdaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbdaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:49:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.646+0000 7fbbdaffd700 10 monclient: dump: 2023-09-30T00:49:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:45.650 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:45.650 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.647+0000 7fbbe2757700 10 monclient: shutdown 2023-09-30T00:49:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.647+0000 7fbbe2757700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.647+0000 7fbbe2757700 1 -- 172.21.15.40:0/1363300981 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbdc0b1830 msgr2=0x7fbbdc0a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.647+0000 7fbbe2757700 1 --2- 172.21.15.40:0/1363300981 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbdc0b1830 0x7fbbdc0a5fb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0009910 tx=0x7fbbd00091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.647+0000 7fbbe2757700 1 -- 172.21.15.40:0/1363300981 shutdown_connections 2023-09-30T00:49:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.647+0000 7fbbe2757700 1 --2- 172.21.15.40:0/1363300981 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbbdc13ac20 0x7fbbdc13b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.647+0000 7fbbe2757700 1 --2- 172.21.15.40:0/1363300981 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbdc0b1830 0x7fbbdc0a5fb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.647+0000 7fbbe2757700 1 -- 172.21.15.40:0/1363300981 >> 172.21.15.40:0/1363300981 conn(0x7fbbdc0631f0 msgr2=0x7fbbdc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.647+0000 7fbbe2757700 1 -- 172.21.15.40:0/1363300981 shutdown_connections 2023-09-30T00:49:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.647+0000 7fbbe2757700 1 -- 172.21.15.40:0/1363300981 wait complete. 2023-09-30T00:49:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.648+0000 7fbbe2757700 10 monclient: build_initial_monmap 2023-09-30T00:49:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.648+0000 7fbbe2757700 10 monclient: monmap: 2023-09-30T00:49:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:45.654 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:45.655 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.648+0000 7fbbe2757700 1 librados: starting msgr at 2023-09-30T00:49:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.648+0000 7fbbe2757700 1 librados: starting objecter 2023-09-30T00:49:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.648+0000 7fbbe2757700 1 Processor -- start 2023-09-30T00:49:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 1 -- start start 2023-09-30T00:49:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 1 librados: setting wanted keys 2023-09-30T00:49:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 1 librados: calling monclient init 2023-09-30T00:49:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 10 monclient: init 2023-09-30T00:49:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:49:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbdc0b1460 0x7fbbdc13df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 10 monclient(hunting): picked mon.noname-a con 0x7fbbdc0b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:45.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbbdc0b1830 0x7fbbdc0a7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 10 monclient(hunting): picked mon.noname-c con 0x7fbbdc0b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbdc13ac20 0x7fbbdc0a9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 10 monclient(hunting): picked mon.noname-b con 0x7fbbdc13ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbdc0a8950 con 0x7fbbdc0b1460 2023-09-30T00:49:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbdc0ab690 con 0x7fbbdc13ac20 2023-09-30T00:49:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbdc0a8390 con 0x7fbbdc0b1830 2023-09-30T00:49:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.649+0000 7fbbe2757700 10 monclient(hunting): authenticate will time out at 5446.950696s 2023-09-30T00:49:45.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbdbfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbdc0b1460 0x7fbbdc13df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbdb7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbbdc0b1830 0x7fbbdc0a7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbe0cf4700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbdc13ac20 0x7fbbdc0a9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbdbfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbdc0b1460 0x7fbbdc13df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56208/0 (socket says 172.21.15.40:56208) 2023-09-30T00:49:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbdbfff700 1 -- 172.21.15.40:0/1520574379 learned_addr learned my addr 172.21.15.40:0/1520574379 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbdbfff700 10 monclient(hunting): get_auth_request con 0x7fbbdc0b1460 auth_method 0 2023-09-30T00:49:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbdbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbdbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbdbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbdb7fe700 10 monclient(hunting): get_auth_request con 0x7fbbdc0b1830 auth_method 0 2023-09-30T00:49:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbdb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbdb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:45.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbdb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:45.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbe0cf4700 10 monclient(hunting): get_auth_request con 0x7fbbdc13ac20 auth_method 0 2023-09-30T00:49:45.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbe0cf4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:45.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbe0cf4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:45.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.650+0000 7fbbe0cf4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:45.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.651+0000 7fbbdb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:45.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.651+0000 7fbbdb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:45.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.651+0000 7fbbdb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:45.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.651+0000 7fbbdbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:45.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.651+0000 7fbbdbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:45.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.651+0000 7fbbdbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:45.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.651+0000 7fbbe0cf4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:45.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.651+0000 7fbbe0cf4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:45.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.651+0000 7fbbe0cf4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:45.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.655+0000 7fbbdb7fe700 10 monclient(hunting): handle_auth_done global_id 4445 payload 824 2023-09-30T00:49:45.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.655+0000 7fbbdb7fe700 1 -- 172.21.15.40:0/1520574379 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbdc13ac20 msgr2=0x7fbbdc0a9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:45.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.655+0000 7fbbdb7fe700 1 --2- 172.21.15.40:0/1520574379 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbdc13ac20 0x7fbbdc0a9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:45.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.655+0000 7fbbdb7fe700 1 -- 172.21.15.40:0/1520574379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbdc0b1460 msgr2=0x7fbbdc13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.655+0000 7fbbdb7fe700 1 --2- 172.21.15.40:0/1520574379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbdc0b1460 0x7fbbdc13df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.655+0000 7fbbdb7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:49:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.655+0000 7fbbdb7fe700 1 monclient: found mon.noname-c 2023-09-30T00:49:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.655+0000 7fbbdb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.655+0000 7fbbdb7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:49:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.655+0000 7fbbdb7fe700 1 -- 172.21.15.40:0/1520574379 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd0009420 con 0x7fbbdc0b1830 2023-09-30T00:49:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.657+0000 7fbbdb7fe700 1 --2- 172.21.15.40:0/1520574379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbbdc0b1830 0x7fbbdc0a7e50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0016000 tx=0x7fbbd0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.657+0000 7fbbc3fff700 10 client.?.objecter ms_handle_connect 0x7fbbdc0b1830 2023-09-30T00:49:45.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.657+0000 7fbbc3fff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:45.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbc3fff700 1 -- 172.21.15.40:0/1520574379 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0000bc0 con 0x7fbbdc0b1830 2023-09-30T00:49:45.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbc3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:45.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbc3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:45.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbc3fff700 10 monclient: dump: 2023-09-30T00:49:45.664 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:45.664 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:45.664 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:45.664 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:45.664 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:45.665 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:45.665 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:45.665 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:45.665 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:45.665 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:45.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbc3fff700 10 monclient: _finish_auth 0 2023-09-30T00:49:45.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbc3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:45.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbc3fff700 1 -- 172.21.15.40:0/1520574379 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd0011900 con 0x7fbbdc0b1830 2023-09-30T00:49:45.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbc3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:45.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbc3fff700 1 -- 172.21.15.40:0/1520574379 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0025420 con 0x7fbbdc0b1830 2023-09-30T00:49:45.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbc3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:45.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbc3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:45.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbc3fff700 10 monclient: dump: 2023-09-30T00:49:45.667 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:45.667 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:45.667 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:45.668 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:45.668 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:45.668 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:45.668 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:45.668 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:45.669 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:45.669 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:45.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbe2757700 5 monclient: authenticate success, global_id 4445 2023-09-30T00:49:45.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbe2757700 10 monclient: _renew_subs 2023-09-30T00:49:45.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbe2757700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:45.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbe2757700 1 -- 172.21.15.40:0/1520574379 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbdc0a8610 con 0x7fbbdc0b1830 2023-09-30T00:49:45.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbda7fc700 4 set_mon_vals no callback set 2023-09-30T00:49:45.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbe2757700 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:45.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbe2757700 10 monclient: _renew_subs 2023-09-30T00:49:45.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbe2757700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:45.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbe2757700 1 -- 172.21.15.40:0/1520574379 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbdc0a9a90 con 0x7fbbdc0b1830 2023-09-30T00:49:45.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.662+0000 7fbbe2757700 1 librados: init done 2023-09-30T00:49:45.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.663+0000 7fbbe2757700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:45.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.663+0000 7fbbe2757700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:45.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.663+0000 7fbbe2757700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:45.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.663+0000 7fbbe2757700 1 -- 172.21.15.40:0/1520574379 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbdc0a9030 con 0x7fbbdc0b1830 2023-09-30T00:49:45.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.977+0000 7fbbc3fff700 1 -- 172.21.15.40:0/1520574379 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fbbd0023070 con 0x7fbbdc0b1830 2023-09-30T00:49:45.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.977+0000 7fbbc3fff700 1 --2- 172.21.15.40:0/1520574379 >> v2:172.21.15.135:6800/116426 conn(0x7fbbc80689e0 0x7fbbc806ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:45.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.978+0000 7fbbc3fff700 1 -- 172.21.15.40:0/1520574379 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fbbd00854b0 con 0x7fbbdc0b1830 2023-09-30T00:49:45.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.978+0000 7fbbc3fff700 10 client.4445.objecter ms_dispatch 0x7fbbdc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:45.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.978+0000 7fbbc3fff700 3 client.4445.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:45.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.978+0000 7fbbc3fff700 3 client.4445.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:45.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.978+0000 7fbbc3fff700 20 client.4445.objecter dump_active .. 0 homeless 2023-09-30T00:49:45.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.978+0000 7fbbc3fff700 1 -- 172.21.15.40:0/1520574379 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbbd0085940 con 0x7fbbdc0b1830 2023-09-30T00:49:45.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.978+0000 7fbbc3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:45.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.978+0000 7fbbc3fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:45.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.978+0000 7fbbdbfff700 1 --2- 172.21.15.40:0/1520574379 >> v2:172.21.15.135:6800/116426 conn(0x7fbbc80689e0 0x7fbbc806ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:45.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.978+0000 7fbbdbfff700 10 monclient: get_auth_request con 0x7fbbc80689e0 auth_method 0 2023-09-30T00:49:45.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.978+0000 7fbbc3fff700 10 client.4445.objecter ms_handle_connect 0x7fbbc80689e0 2023-09-30T00:49:45.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:45.978+0000 7fbbdbfff700 1 --2- 172.21.15.40:0/1520574379 >> v2:172.21.15.135:6800/116426 conn(0x7fbbc80689e0 0x7fbbc806ae90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc007fd0 tx=0x7fbbcc000bc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:46.253 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.250+0000 7fbbe2757700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:46.253 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.250+0000 7fbbe2757700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:46.253 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.250+0000 7fbbe2757700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:46.253 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.250+0000 7fbbe2757700 1 -- 172.21.15.40:0/1520574379 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbbdc0a9030 con 0x7fbbdc0b1830 2023-09-30T00:49:46.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.258+0000 7fbbc3fff700 1 -- 172.21.15.40:0/1520574379 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbbdc0a9030 con 0x7fbbdc0b1830 2023-09-30T00:49:46.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.258+0000 7fbbc3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:46.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.258+0000 7fbbc3fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:46.259 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:46.259 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:46.261 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 10 librados: watch_flush enter 2023-09-30T00:49:46.261 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 10 librados: watch_flush exit 2023-09-30T00:49:46.261 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 20 client.4445.objecter shutdown clearing up homeless session... 2023-09-30T00:49:46.261 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 10 client.4445.objecter successfully canceled tick 2023-09-30T00:49:46.261 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 1 -- 172.21.15.40:0/1520574379 >> v2:172.21.15.135:6800/116426 conn(0x7fbbc80689e0 msgr2=0x7fbbc806ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:46.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 1 --2- 172.21.15.40:0/1520574379 >> v2:172.21.15.135:6800/116426 conn(0x7fbbc80689e0 0x7fbbc806ae90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc007fd0 tx=0x7fbbcc000bc0 comp rx=0 tx=0).stop 2023-09-30T00:49:46.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 10 monclient: shutdown 2023-09-30T00:49:46.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:46.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 1 -- 172.21.15.40:0/1520574379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbbdc0b1830 msgr2=0x7fbbdc0a7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:46.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 1 --2- 172.21.15.40:0/1520574379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbbdc0b1830 0x7fbbdc0a7e50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0016000 tx=0x7fbbd0017000 comp rx=0 tx=0).stop 2023-09-30T00:49:46.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 1 -- 172.21.15.40:0/1520574379 shutdown_connections 2023-09-30T00:49:46.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 1 --2- 172.21.15.40:0/1520574379 >> v2:172.21.15.135:6800/116426 conn(0x7fbbc80689e0 0x7fbbc806ae90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:46.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 1 --2- 172.21.15.40:0/1520574379 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbbdc13ac20 0x7fbbdc0a9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:46.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 1 --2- 172.21.15.40:0/1520574379 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbdc0b1460 0x7fbbdc13df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:46.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 1 --2- 172.21.15.40:0/1520574379 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbbdc0b1830 0x7fbbdc0a7e50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:46.263 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 1 -- 172.21.15.40:0/1520574379 >> 172.21.15.40:0/1520574379 conn(0x7fbbdc0631f0 msgr2=0x7fbbdc0b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:46.263 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 1 -- 172.21.15.40:0/1520574379 shutdown_connections 2023-09-30T00:49:46.263 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 1 -- 172.21.15.40:0/1520574379 wait complete. 2023-09-30T00:49:46.263 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:46.260+0000 7fbbe2757700 1 librados: shutdown 2023-09-30T00:49:48.068 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:03:23.779 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:48.127 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:23.813 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:48.133 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:23.820 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:48.141 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:03:23.852 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:48.161 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:23.848 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:48.171 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:03:23.882 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:48.193 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:03:23.905 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:48.200 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:03:23.912 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:48.231 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:03:23.943 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:49:49.276 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.411+0000 7f9d6e84b700 10 monclient: get_monmap_and_config 2023-09-30T00:49:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.411+0000 7f9d6e84b700 10 monclient: build_initial_monmap 2023-09-30T00:49:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.411+0000 7f9d6e84b700 10 monclient: monmap: 2023-09-30T00:49:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:49.412754+0000 2023-09-30T00:49:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:49.412754+0000 2023-09-30T00:49:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:49.414 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.412+0000 7f9d6e84b700 1 Processor -- start 2023-09-30T00:49:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.412+0000 7f9d6e84b700 1 -- start start 2023-09-30T00:49:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.412+0000 7f9d6e84b700 10 monclient: init 2023-09-30T00:49:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-09-30T00:49:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9d680b1460 0x7f9d680b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 10 monclient(hunting): picked mon.noname-a con 0x7f9d680b1460 addr v2:172.21.15.40:3300/0 2023-09-30T00:49:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d680a5fb0 0x7f9d68141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 10 monclient(hunting): picked mon.noname-b con 0x7f9d680a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9d680af410 0x7f9d680af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 10 monclient(hunting): picked mon.noname-c con 0x7f9d680af410 addr v2:172.21.15.135:3300/0 2023-09-30T00:49:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d680a71f0 con 0x7f9d680b1460 2023-09-30T00:49:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d680a74d0 con 0x7f9d680af410 2023-09-30T00:49:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d680a63c0 con 0x7f9d680a5fb0 2023-09-30T00:49:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.413+0000 7f9d6e84b700 10 monclient(hunting): authenticate will time out at 5450.714629s 2023-09-30T00:49:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.414+0000 7f9d67fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9d680b1460 0x7f9d680b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.414+0000 7f9d6cde8700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9d680af410 0x7f9d680af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.414+0000 7f9d677fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d680a5fb0 0x7f9d68141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.414+0000 7f9d67fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9d680b1460 0x7f9d680b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56214/0 (socket says 172.21.15.40:56214) 2023-09-30T00:49:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.414+0000 7f9d67fff700 1 -- 172.21.15.40:0/3961167926 learned_addr learned my addr 172.21.15.40:0/3961167926 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.414+0000 7f9d67fff700 10 monclient(hunting): get_auth_request con 0x7f9d680b1460 auth_method 0 2023-09-30T00:49:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.414+0000 7f9d67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.414+0000 7f9d67fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.414+0000 7f9d67fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.414+0000 7f9d6cde8700 10 monclient(hunting): get_auth_request con 0x7f9d680af410 auth_method 0 2023-09-30T00:49:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.414+0000 7f9d6cde8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.414+0000 7f9d6cde8700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.414+0000 7f9d6cde8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.415+0000 7f9d677fe700 10 monclient(hunting): get_auth_request con 0x7f9d680a5fb0 auth_method 0 2023-09-30T00:49:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.415+0000 7f9d677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.415+0000 7f9d677fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.415+0000 7f9d677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.415+0000 7f9d67fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.415+0000 7f9d67fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.415+0000 7f9d67fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:49.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.415+0000 7f9d6cde8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.415+0000 7f9d6cde8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.415+0000 7f9d6cde8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.415+0000 7f9d677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.415+0000 7f9d677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:49.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.415+0000 7f9d677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:49.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.418+0000 7f9d6cde8700 10 monclient(hunting): handle_auth_done global_id 4555 payload 274 2023-09-30T00:49:49.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.418+0000 7f9d6cde8700 1 -- 172.21.15.40:0/3961167926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d680a5fb0 msgr2=0x7f9d68141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:49.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.418+0000 7f9d6cde8700 1 --2- 172.21.15.40:0/3961167926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d680a5fb0 0x7f9d68141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:49.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.418+0000 7f9d6cde8700 1 -- 172.21.15.40:0/3961167926 >> v2:172.21.15.40:3300/0 conn(0x7f9d680b1460 msgr2=0x7f9d680b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:49.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.418+0000 7f9d6cde8700 1 --2- 172.21.15.40:0/3961167926 >> v2:172.21.15.40:3300/0 conn(0x7f9d680b1460 0x7f9d680b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:49.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.418+0000 7f9d6cde8700 10 monclient: _finish_hunting 0 2023-09-30T00:49:49.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.418+0000 7f9d6cde8700 1 monclient: found mon.noname-c 2023-09-30T00:49:49.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.418+0000 7f9d6cde8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:49.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.418+0000 7f9d6cde8700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:49:49.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.418+0000 7f9d6cde8700 1 -- 172.21.15.40:0/3961167926 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d680a56d0 con 0x7f9d680af410 2023-09-30T00:49:49.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.420+0000 7f9d6cde8700 1 --2- 172.21.15.40:0/3961167926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d680af410 0x7f9d680af890 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9d680b05f0 tx=0x7f9d60009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93e098282ccad6ca server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:49.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d66ffd700 1 -- 172.21.15.40:0/3961167926 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6000e070 con 0x7f9d680af410 2023-09-30T00:49:49.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:49.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d66ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:49.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d66ffd700 10 monclient: dump: 2023-09-30T00:49:49.426 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:49.426 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:49.426 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:49.426 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:49.426 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:49.426 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:49.427 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:49.427 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:49.427 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:49.427 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:49.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d66ffd700 10 monclient: _finish_auth 0 2023-09-30T00:49:49.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d66ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:49.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d66ffd700 1 -- 172.21.15.40:0/3961167926 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d6000bab0 con 0x7f9d680af410 2023-09-30T00:49:49.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d66ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:49.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d66ffd700 1 -- 172.21.15.40:0/3961167926 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d6000be70 con 0x7f9d680af410 2023-09-30T00:49:49.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:49.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d66ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:49:49.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d66ffd700 10 monclient: dump: 2023-09-30T00:49:49.428 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:49.428 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:49.428 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:49.428 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:49.428 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:49.428 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:49.428 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:49.428 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:49.428 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:49.428 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:49.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d6e84b700 5 monclient: authenticate success, global_id 4555 2023-09-30T00:49:49.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d6e84b700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:49.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.425+0000 7f9d6e84b700 4 set_mon_vals no callback set 2023-09-30T00:49:49.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.426+0000 7f9d6e84b700 10 monclient: shutdown 2023-09-30T00:49:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.426+0000 7f9d6e84b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.426+0000 7f9d6e84b700 1 -- 172.21.15.40:0/3961167926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d680af410 msgr2=0x7f9d680af890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.426+0000 7f9d6e84b700 1 --2- 172.21.15.40:0/3961167926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d680af410 0x7f9d680af890 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9d680b05f0 tx=0x7f9d60009780 comp rx=0 tx=0).stop 2023-09-30T00:49:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.426+0000 7f9d6e84b700 1 -- 172.21.15.40:0/3961167926 shutdown_connections 2023-09-30T00:49:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.426+0000 7f9d6e84b700 1 --2- 172.21.15.40:0/3961167926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d680af410 0x7f9d680af890 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.426+0000 7f9d6e84b700 1 --2- 172.21.15.40:0/3961167926 >> v2:172.21.15.40:3300/0 conn(0x7f9d680b1460 0x7f9d680b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.426+0000 7f9d6e84b700 1 --2- 172.21.15.40:0/3961167926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d680a5fb0 0x7f9d68141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.426+0000 7f9d6e84b700 1 -- 172.21.15.40:0/3961167926 >> 172.21.15.40:0/3961167926 conn(0x7f9d680631f0 msgr2=0x7f9d680635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.426+0000 7f9d6e84b700 1 -- 172.21.15.40:0/3961167926 shutdown_connections 2023-09-30T00:49:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.426+0000 7f9d6e84b700 1 -- 172.21.15.40:0/3961167926 wait complete. 2023-09-30T00:49:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.427+0000 7f9d6e84b700 10 monclient: build_initial_monmap 2023-09-30T00:49:49.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.427+0000 7f9d6e84b700 10 monclient: monmap: 2023-09-30T00:49:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:49.433 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.427+0000 7f9d6e84b700 1 librados: starting msgr at 2023-09-30T00:49:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.427+0000 7f9d6e84b700 1 librados: starting objecter 2023-09-30T00:49:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.427+0000 7f9d6e84b700 1 Processor -- start 2023-09-30T00:49:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.427+0000 7f9d6e84b700 1 -- start start 2023-09-30T00:49:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.427+0000 7f9d6e84b700 1 librados: setting wanted keys 2023-09-30T00:49:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.427+0000 7f9d6e84b700 1 librados: calling monclient init 2023-09-30T00:49:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.427+0000 7f9d6e84b700 10 monclient: init 2023-09-30T00:49:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:49:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d680a5fb0 0x7f9d68136890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 10 monclient(hunting): picked mon.noname-c con 0x7f9d680a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d680af410 0x7f9d68144930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 10 monclient(hunting): picked mon.noname-b con 0x7f9d680af410 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d680b1460 0x7f9d68144e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 10 monclient(hunting): picked mon.noname-a con 0x7f9d680b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d68150b80 con 0x7f9d680b1460 2023-09-30T00:49:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d681453b0 con 0x7f9d680af410 2023-09-30T00:49:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6814ec40 con 0x7f9d680a5fb0 2023-09-30T00:49:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.428+0000 7f9d6e84b700 10 monclient(hunting): authenticate will time out at 5450.729448s 2023-09-30T00:49:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d677fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d680af410 0x7f9d68144930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:49.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d6cde8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d680b1460 0x7f9d68144e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d67fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d680a5fb0 0x7f9d68136890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d677fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d680af410 0x7f9d68144930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:57884/0 (socket says 172.21.15.40:57884) 2023-09-30T00:49:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d677fe700 1 -- 172.21.15.40:0/1989853349 learned_addr learned my addr 172.21.15.40:0/1989853349 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d677fe700 10 monclient(hunting): get_auth_request con 0x7f9d680af410 auth_method 0 2023-09-30T00:49:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d6cde8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d680b1460 0x7f9d68144e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56228/0 (socket says 172.21.15.40:56228) 2023-09-30T00:49:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d677fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d6cde8700 10 monclient(hunting): get_auth_request con 0x7f9d680b1460 auth_method 0 2023-09-30T00:49:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d6cde8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d6cde8700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:49.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d6cde8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d67fff700 10 monclient(hunting): get_auth_request con 0x7f9d680a5fb0 auth_method 0 2023-09-30T00:49:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d67fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d67fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d6cde8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d6cde8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.429+0000 7f9d6cde8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.430+0000 7f9d67fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.430+0000 7f9d67fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:49.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.430+0000 7f9d67fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.433+0000 7f9d677fe700 10 monclient(hunting): handle_auth_done global_id 4558 payload 824 2023-09-30T00:49:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.433+0000 7f9d677fe700 1 -- 172.21.15.40:0/1989853349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d680a5fb0 msgr2=0x7f9d68136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:49.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.433+0000 7f9d677fe700 1 --2- 172.21.15.40:0/1989853349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d680a5fb0 0x7f9d68136890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:49.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.433+0000 7f9d677fe700 1 -- 172.21.15.40:0/1989853349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d680b1460 msgr2=0x7f9d68144e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:49.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.433+0000 7f9d677fe700 1 --2- 172.21.15.40:0/1989853349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d680b1460 0x7f9d68144e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:49.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.433+0000 7f9d677fe700 10 monclient: _finish_hunting 0 2023-09-30T00:49:49.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.433+0000 7f9d677fe700 1 monclient: found mon.noname-b 2023-09-30T00:49:49.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.433+0000 7f9d677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:49.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.433+0000 7f9d677fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:49:49.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.433+0000 7f9d677fe700 1 -- 172.21.15.40:0/1989853349 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d60009a20 con 0x7f9d680af410 2023-09-30T00:49:49.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.435+0000 7f9d677fe700 1 --2- 172.21.15.40:0/1989853349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d680af410 0x7f9d68144930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c00ec80 tx=0x7f9d5c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:49.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.435+0000 7f9d4ffff700 10 client.?.objecter ms_handle_connect 0x7f9d680af410 2023-09-30T00:49:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.435+0000 7f9d4ffff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d4ffff700 1 -- 172.21.15.40:0/1989853349 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c009690 con 0x7f9d680af410 2023-09-30T00:49:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d4ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d4ffff700 10 monclient: dump: 2023-09-30T00:49:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:49.441 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:49.442 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:49.442 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:49.442 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:49.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d4ffff700 10 monclient: _finish_auth 0 2023-09-30T00:49:49.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d4ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:49.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d6e84b700 5 monclient: authenticate success, global_id 4558 2023-09-30T00:49:49.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d6e84b700 10 monclient: _renew_subs 2023-09-30T00:49:49.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d6e84b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:49.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d6e84b700 1 -- 172.21.15.40:0/1989853349 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d6813ed90 con 0x7f9d680af410 2023-09-30T00:49:49.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d6e84b700 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:49.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d6e84b700 10 monclient: _renew_subs 2023-09-30T00:49:49.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d6e84b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:49.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d6e84b700 1 -- 172.21.15.40:0/1989853349 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6813cf60 con 0x7f9d680af410 2023-09-30T00:49:49.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.437+0000 7f9d6e84b700 1 librados: init done 2023-09-30T00:49:49.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.438+0000 7f9d6e84b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:49.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.438+0000 7f9d6e84b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:49.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.438+0000 7f9d6e84b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:49.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.438+0000 7f9d6e84b700 1 -- 172.21.15.40:0/1989853349 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d68064a00 con 0x7f9d680af410 2023-09-30T00:49:49.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.439+0000 7f9d4ffff700 1 -- 172.21.15.40:0/1989853349 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5c00ca60 con 0x7f9d680af410 2023-09-30T00:49:49.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.439+0000 7f9d4ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:49.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.439+0000 7f9d4ffff700 1 -- 172.21.15.40:0/1989853349 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c00cbd0 con 0x7f9d680af410 2023-09-30T00:49:49.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.439+0000 7f9d4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:49.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.439+0000 7f9d4ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:49:49.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.439+0000 7f9d4ffff700 10 monclient: dump: 2023-09-30T00:49:49.444 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:49.444 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:49.444 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:49.444 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:49.444 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:49.444 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:49.444 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:49.444 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:49.444 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:49.444 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:49.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.439+0000 7f9d667fc700 4 set_mon_vals no callback set 2023-09-30T00:49:49.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.659+0000 7f9d4ffff700 1 -- 172.21.15.40:0/1989853349 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9d5c00cd30 con 0x7f9d680af410 2023-09-30T00:49:49.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.660+0000 7f9d4ffff700 1 --2- 172.21.15.40:0/1989853349 >> v2:172.21.15.135:6800/116426 conn(0x7f9d54068890 0x7f9d5406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:49.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.660+0000 7f9d67fff700 1 --2- 172.21.15.40:0/1989853349 >> v2:172.21.15.135:6800/116426 conn(0x7f9d54068890 0x7f9d5406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:49.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.660+0000 7f9d67fff700 10 monclient: get_auth_request con 0x7f9d54068890 auth_method 0 2023-09-30T00:49:49.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.660+0000 7f9d67fff700 1 --2- 172.21.15.40:0/1989853349 >> v2:172.21.15.135:6800/116426 conn(0x7f9d54068890 0x7f9d5406ad40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d58006fd0 tx=0x7f9d58009250 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:49.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.660+0000 7f9d4ffff700 10 client.4558.objecter ms_handle_connect 0x7f9d54068890 2023-09-30T00:49:49.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.700+0000 7f9d4ffff700 1 -- 172.21.15.40:0/1989853349 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9d5c056310 con 0x7f9d680af410 2023-09-30T00:49:49.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.700+0000 7f9d4ffff700 10 client.4558.objecter ms_dispatch 0x7f9d68001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:49.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.700+0000 7f9d4ffff700 3 client.4558.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:49.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.700+0000 7f9d4ffff700 3 client.4558.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:49.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.700+0000 7f9d4ffff700 20 client.4558.objecter dump_active .. 0 homeless 2023-09-30T00:49:49.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.709+0000 7f9d4ffff700 1 -- 172.21.15.40:0/1989853349 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9d5c051f50 con 0x7f9d680af410 2023-09-30T00:49:49.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.709+0000 7f9d4ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:49.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.709+0000 7f9d4ffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.970+0000 7f9d6e84b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.970+0000 7f9d6e84b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.970+0000 7f9d6e84b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:49.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.970+0000 7f9d6e84b700 1 -- 172.21.15.40:0/1989853349 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9d680b4570 con 0x7f9d680af410 2023-09-30T00:49:49.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.976+0000 7f9d4ffff700 1 -- 172.21.15.40:0/1989853349 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9d5c017100 con 0x7f9d680af410 2023-09-30T00:49:49.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.976+0000 7f9d4ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:49.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.976+0000 7f9d4ffff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:49.978 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:49.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 10 librados: watch_flush enter 2023-09-30T00:49:49.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 10 librados: watch_flush exit 2023-09-30T00:49:49.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 20 client.4558.objecter shutdown clearing up homeless session... 2023-09-30T00:49:49.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 10 client.4558.objecter successfully canceled tick 2023-09-30T00:49:49.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 1 -- 172.21.15.40:0/1989853349 >> v2:172.21.15.135:6800/116426 conn(0x7f9d54068890 msgr2=0x7f9d5406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:49.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 1 --2- 172.21.15.40:0/1989853349 >> v2:172.21.15.135:6800/116426 conn(0x7f9d54068890 0x7f9d5406ad40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d58006fd0 tx=0x7f9d58009250 comp rx=0 tx=0).stop 2023-09-30T00:49:49.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 10 monclient: shutdown 2023-09-30T00:49:49.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:49.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 1 -- 172.21.15.40:0/1989853349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d680af410 msgr2=0x7f9d68144930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:49.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 1 --2- 172.21.15.40:0/1989853349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d680af410 0x7f9d68144930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c00ec80 tx=0x7f9d5c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:49.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 1 -- 172.21.15.40:0/1989853349 shutdown_connections 2023-09-30T00:49:49.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 1 --2- 172.21.15.40:0/1989853349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d680b1460 0x7f9d68144e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:49.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 1 --2- 172.21.15.40:0/1989853349 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9d680a5fb0 0x7f9d68136890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:49.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 1 --2- 172.21.15.40:0/1989853349 >> v2:172.21.15.135:6800/116426 conn(0x7f9d54068890 0x7f9d5406ad40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:49.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 1 --2- 172.21.15.40:0/1989853349 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9d680af410 0x7f9d68144930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:49.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 1 -- 172.21.15.40:0/1989853349 >> 172.21.15.40:0/1989853349 conn(0x7f9d680631f0 msgr2=0x7f9d680b28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:49.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 1 -- 172.21.15.40:0/1989853349 shutdown_connections 2023-09-30T00:49:49.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 1 -- 172.21.15.40:0/1989853349 wait complete. 2023-09-30T00:49:49.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:49.979+0000 7f9d6e84b700 1 librados: shutdown 2023-09-30T00:49:52.995 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:53.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.137+0000 7f7c9b24f700 10 monclient: get_monmap_and_config 2023-09-30T00:49:53.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.137+0000 7f7c9b24f700 10 monclient: build_initial_monmap 2023-09-30T00:49:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.137+0000 7f7c9b24f700 10 monclient: monmap: 2023-09-30T00:49:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:53.138958+0000 2023-09-30T00:49:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:53.138958+0000 2023-09-30T00:49:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:53.140 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:53.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.138+0000 7f7c9b24f700 1 Processor -- start 2023-09-30T00:49:53.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.138+0000 7f7c9b24f700 1 -- start start 2023-09-30T00:49:53.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.138+0000 7f7c9b24f700 10 monclient: init 2023-09-30T00:49:53.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.139+0000 7f7c9b24f700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:53.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.139+0000 7f7c9b24f700 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-09-30T00:49:53.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.139+0000 7f7c9b24f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c940b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:49:53.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.139+0000 7f7c9b24f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c940b1830 0x7f7c940a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:53.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.139+0000 7f7c9b24f700 10 monclient(hunting): picked mon.noname-b con 0x7f7c940b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:53.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.139+0000 7f7c9b24f700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7c9413ac20 0x7f7c9413b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:53.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.139+0000 7f7c9b24f700 10 monclient(hunting): picked mon.noname-c con 0x7f7c9413ac20 addr v2:172.21.15.135:3300/0 2023-09-30T00:49:53.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.139+0000 7f7c9b24f700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c940a5b20 con 0x7f7c940b1460 2023-09-30T00:49:53.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.139+0000 7f7c9b24f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:53.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.139+0000 7f7c9b24f700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c94141e40 con 0x7f7c9413ac20 2023-09-30T00:49:53.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.139+0000 7f7c9b24f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:53.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.139+0000 7f7c9b24f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c940a64f0 con 0x7f7c940b1830 2023-09-30T00:49:53.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.139+0000 7f7c9b24f700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.140+0000 7f7c9b24f700 10 monclient(hunting): authenticate will time out at 5454.441007s 2023-09-30T00:49:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.140+0000 7f7c93fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c940b1830 0x7f7c940a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.140+0000 7f7c997ec700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7c9413ac20 0x7f7c9413b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c93fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c940b1830 0x7f7c940a5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:58208/0 (socket says 172.21.15.40:58208) 2023-09-30T00:49:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c93fff700 1 -- 172.21.15.40:0/3242013137 learned_addr learned my addr 172.21.15.40:0/3242013137 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c93fff700 10 monclient(hunting): get_auth_request con 0x7f7c940b1830 auth_method 0 2023-09-30T00:49:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c93fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c93fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c93fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c997ec700 10 monclient(hunting): get_auth_request con 0x7f7c9413ac20 auth_method 0 2023-09-30T00:49:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c997ec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c997ec700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:53.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c997ec700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:53.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c997ec700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c997ec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c997ec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c93fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.141+0000 7f7c93fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.142+0000 7f7c93fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.144+0000 7f7c93fff700 10 monclient(hunting): handle_auth_done global_id 4454 payload 274 2023-09-30T00:49:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.144+0000 7f7c93fff700 1 -- 172.21.15.40:0/3242013137 >> v2:172.21.15.135:3300/0 conn(0x7f7c9413ac20 msgr2=0x7f7c9413b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.144+0000 7f7c93fff700 1 --2- 172.21.15.40:0/3242013137 >> v2:172.21.15.135:3300/0 conn(0x7f7c9413ac20 0x7f7c9413b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.144+0000 7f7c93fff700 1 -- 172.21.15.40:0/3242013137 >> v1:172.21.15.135:6789/0 conn(0x7f7c940b1460 legacy=0x7f7c94141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.144+0000 7f7c93fff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:53.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.144+0000 7f7c93fff700 1 monclient: found mon.noname-b 2023-09-30T00:49:53.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.144+0000 7f7c93fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:53.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.144+0000 7f7c93fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:49:53.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.144+0000 7f7c93fff700 1 -- 172.21.15.40:0/3242013137 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c940a71f0 con 0x7f7c940b1830 2023-09-30T00:49:53.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.146+0000 7f7c93fff700 1 --2- 172.21.15.40:0/3242013137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c940b1830 0x7f7c940a5fb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7c880099b0 tx=0x7f7c88009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=522b417162ff5a31 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:53.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c937fe700 1 -- 172.21.15.40:0/3242013137 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c88009ea0 con 0x7f7c940b1830 2023-09-30T00:49:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c937fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c937fe700 10 monclient: dump: 2023-09-30T00:49:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:53.153 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c937fe700 10 monclient: _finish_auth 0 2023-09-30T00:49:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c937fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c937fe700 1 -- 172.21.15.40:0/3242013137 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8800b820 con 0x7f7c940b1830 2023-09-30T00:49:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c9b24f700 5 monclient: authenticate success, global_id 4454 2023-09-30T00:49:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c9b24f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c937fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c9b24f700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c937fe700 1 -- 172.21.15.40:0/3242013137 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8800bbc0 con 0x7f7c940b1830 2023-09-30T00:49:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c9b24f700 4 set_mon_vals no callback set 2023-09-30T00:49:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c937fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c937fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.151+0000 7f7c937fe700 10 monclient: dump: 2023-09-30T00:49:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:53.154 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:53.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.152+0000 7f7c9b24f700 10 monclient: shutdown 2023-09-30T00:49:53.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.152+0000 7f7c9b24f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:53.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.152+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3242013137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c940b1830 msgr2=0x7f7c940a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:53.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.152+0000 7f7c9b24f700 1 --2- 172.21.15.40:0/3242013137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c940b1830 0x7f7c940a5fb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7c880099b0 tx=0x7f7c88009250 comp rx=0 tx=0).stop 2023-09-30T00:49:53.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.152+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3242013137 shutdown_connections 2023-09-30T00:49:53.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.152+0000 7f7c9b24f700 1 --2- 172.21.15.40:0/3242013137 >> v2:172.21.15.135:3300/0 conn(0x7f7c9413ac20 0x7f7c9413b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:53.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.152+0000 7f7c9b24f700 1 --2- 172.21.15.40:0/3242013137 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c940b1830 0x7f7c940a5fb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:53.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.152+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3242013137 >> 172.21.15.40:0/3242013137 conn(0x7f7c940631f0 msgr2=0x7f7c940635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:53.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.152+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3242013137 shutdown_connections 2023-09-30T00:49:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.152+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3242013137 wait complete. 2023-09-30T00:49:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.153+0000 7f7c9b24f700 10 monclient: build_initial_monmap 2023-09-30T00:49:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.153+0000 7f7c9b24f700 10 monclient: monmap: 2023-09-30T00:49:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:53.159 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.153+0000 7f7c9b24f700 1 librados: starting msgr at 2023-09-30T00:49:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.153+0000 7f7c9b24f700 1 librados: starting objecter 2023-09-30T00:49:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.154+0000 7f7c9b24f700 1 Processor -- start 2023-09-30T00:49:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.154+0000 7f7c9b24f700 1 -- start start 2023-09-30T00:49:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.154+0000 7f7c9b24f700 1 librados: setting wanted keys 2023-09-30T00:49:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.154+0000 7f7c9b24f700 1 librados: calling monclient init 2023-09-30T00:49:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.154+0000 7f7c9b24f700 10 monclient: init 2023-09-30T00:49:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:49:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c940b1460 0x7f7c940a9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 10 monclient(hunting): picked mon.noname-a con 0x7f7c940b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c940b1830 0x7f7c9413e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 10 monclient(hunting): picked mon.noname-c con 0x7f7c940b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c9413ac20 0x7f7c940aa040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 10 monclient(hunting): picked mon.noname-b con 0x7f7c9413ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9413f070 con 0x7f7c940b1460 2023-09-30T00:49:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c940a9bb0 con 0x7f7c9413ac20 2023-09-30T00:49:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9413e5c0 con 0x7f7c940b1830 2023-09-30T00:49:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.155+0000 7f7c9b24f700 10 monclient(hunting): authenticate will time out at 5454.456238s 2023-09-30T00:49:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.156+0000 7f7c997ec700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c9413ac20 0x7f7c940aa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.156+0000 7f7c997ec700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c9413ac20 0x7f7c940aa040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:51014/0 (socket says 172.21.15.40:51014) 2023-09-30T00:49:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.156+0000 7f7c997ec700 1 -- 172.21.15.40:0/3128555275 learned_addr learned my addr 172.21.15.40:0/3128555275 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.156+0000 7f7c997ec700 10 monclient(hunting): get_auth_request con 0x7f7c9413ac20 auth_method 0 2023-09-30T00:49:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.156+0000 7f7c997ec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.156+0000 7f7c997ec700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.156+0000 7f7c997ec700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.156+0000 7f7c93fff700 1 --2- 172.21.15.40:0/3128555275 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c940b1830 0x7f7c9413e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.156+0000 7f7c997ec700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.156+0000 7f7c997ec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.156+0000 7f7c997ec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.157+0000 7f7c93fff700 10 monclient(hunting): get_auth_request con 0x7f7c940b1830 auth_method 0 2023-09-30T00:49:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.157+0000 7f7c93fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.157+0000 7f7c93fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.157+0000 7f7c93fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.158+0000 7f7c93fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.158+0000 7f7c93fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.158+0000 7f7c93fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.159+0000 7f7c98feb700 1 --2- 172.21.15.40:0/3128555275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c940b1460 0x7f7c940a9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.159+0000 7f7c98feb700 10 monclient(hunting): get_auth_request con 0x7f7c940b1460 auth_method 0 2023-09-30T00:49:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.159+0000 7f7c98feb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.159+0000 7f7c98feb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.159+0000 7f7c98feb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.160+0000 7f7c997ec700 10 monclient(hunting): handle_auth_done global_id 4564 payload 824 2023-09-30T00:49:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.160+0000 7f7c997ec700 1 -- 172.21.15.40:0/3128555275 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c940b1830 msgr2=0x7f7c9413e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.160+0000 7f7c997ec700 1 --2- 172.21.15.40:0/3128555275 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c940b1830 0x7f7c9413e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.160+0000 7f7c997ec700 1 -- 172.21.15.40:0/3128555275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c940b1460 msgr2=0x7f7c940a9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.160+0000 7f7c997ec700 1 --2- 172.21.15.40:0/3128555275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c940b1460 0x7f7c940a9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.160+0000 7f7c997ec700 10 monclient: _finish_hunting 0 2023-09-30T00:49:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.160+0000 7f7c997ec700 1 monclient: found mon.noname-b 2023-09-30T00:49:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.160+0000 7f7c997ec700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.160+0000 7f7c997ec700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:49:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.160+0000 7f7c997ec700 1 -- 172.21.15.40:0/3128555275 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c880094f0 con 0x7f7c9413ac20 2023-09-30T00:49:53.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.163+0000 7f7c997ec700 1 --2- 172.21.15.40:0/3128555275 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c9413ac20 0x7f7c940aa040 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c009df0 tx=0x7f7c8c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:53.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.163+0000 7f7c90ff9700 10 client.?.objecter ms_handle_connect 0x7f7c9413ac20 2023-09-30T00:49:53.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.163+0000 7f7c90ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c90ff9700 1 -- 172.21.15.40:0/3128555275 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c00cb50 con 0x7f7c9413ac20 2023-09-30T00:49:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c90ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c90ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c90ff9700 10 monclient: dump: 2023-09-30T00:49:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:53.220 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c90ff9700 10 monclient: _finish_auth 0 2023-09-30T00:49:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c90ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c9b24f700 5 monclient: authenticate success, global_id 4564 2023-09-30T00:49:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c9b24f700 10 monclient: _renew_subs 2023-09-30T00:49:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c9b24f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3128555275 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c940ae610 con 0x7f7c9413ac20 2023-09-30T00:49:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c9b24f700 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c9b24f700 10 monclient: _renew_subs 2023-09-30T00:49:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c9b24f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3128555275 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c940aa580 con 0x7f7c9413ac20 2023-09-30T00:49:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.218+0000 7f7c9b24f700 1 librados: init done 2023-09-30T00:49:53.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.219+0000 7f7c9b24f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:53.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.219+0000 7f7c9b24f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:53.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.219+0000 7f7c9b24f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:53.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.219+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3128555275 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c940b4610 con 0x7f7c9413ac20 2023-09-30T00:49:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.224+0000 7f7c90ff9700 1 -- 172.21.15.40:0/3128555275 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8c00f040 con 0x7f7c9413ac20 2023-09-30T00:49:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.224+0000 7f7c90ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.224+0000 7f7c90ff9700 1 -- 172.21.15.40:0/3128555275 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c00cec0 con 0x7f7c9413ac20 2023-09-30T00:49:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.224+0000 7f7c90ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.224+0000 7f7c90ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:49:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.224+0000 7f7c90ff9700 10 monclient: dump: 2023-09-30T00:49:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:53.226 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:53.226 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:53.226 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:53.226 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:53.226 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:53.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.224+0000 7f7c92ffd700 4 set_mon_vals no callback set 2023-09-30T00:49:53.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.473+0000 7f7c90ff9700 1 -- 172.21.15.40:0/3128555275 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7c8c014480 con 0x7f7c9413ac20 2023-09-30T00:49:53.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.473+0000 7f7c90ff9700 1 --2- 172.21.15.40:0/3128555275 >> v2:172.21.15.135:6800/116426 conn(0x7f7c80068890 0x7f7c8006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:53.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.473+0000 7f7c98feb700 1 --2- 172.21.15.40:0/3128555275 >> v2:172.21.15.135:6800/116426 conn(0x7f7c80068890 0x7f7c8006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:53.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.473+0000 7f7c98feb700 10 monclient: get_auth_request con 0x7f7c80068890 auth_method 0 2023-09-30T00:49:53.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.474+0000 7f7c90ff9700 10 client.4564.objecter ms_handle_connect 0x7f7c80068890 2023-09-30T00:49:53.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.474+0000 7f7c98feb700 1 --2- 172.21.15.40:0/3128555275 >> v2:172.21.15.135:6800/116426 conn(0x7f7c80068890 0x7f7c8006ad40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7c84006fd0 tx=0x7f7c84008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:53.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.478+0000 7f7c90ff9700 1 -- 172.21.15.40:0/3128555275 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7c8c056980 con 0x7f7c9413ac20 2023-09-30T00:49:53.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.478+0000 7f7c90ff9700 10 client.4564.objecter ms_dispatch 0x7f7c94001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:53.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.478+0000 7f7c90ff9700 3 client.4564.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:53.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.478+0000 7f7c90ff9700 3 client.4564.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:53.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.478+0000 7f7c90ff9700 20 client.4564.objecter dump_active .. 0 homeless 2023-09-30T00:49:53.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.490+0000 7f7c90ff9700 1 -- 172.21.15.40:0/3128555275 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c8c0525c0 con 0x7f7c9413ac20 2023-09-30T00:49:53.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.490+0000 7f7c90ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:53.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.490+0000 7f7c90ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:53.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.740+0000 7f7c9b24f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:53.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.740+0000 7f7c9b24f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:53.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.740+0000 7f7c9b24f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:49:53.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.740+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3128555275 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7c94064a00 con 0x7f7c9413ac20 2023-09-30T00:49:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.744+0000 7f7c90ff9700 1 -- 172.21.15.40:0/3128555275 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7c8c055be0 con 0x7f7c9413ac20 2023-09-30T00:49:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.744+0000 7f7c90ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:53.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.744+0000 7f7c90ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:53.745 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:53.745 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.746+0000 7f7c9b24f700 10 librados: watch_flush enter 2023-09-30T00:49:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.746+0000 7f7c9b24f700 10 librados: watch_flush exit 2023-09-30T00:49:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.746+0000 7f7c9b24f700 20 client.4564.objecter shutdown clearing up homeless session... 2023-09-30T00:49:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.746+0000 7f7c9b24f700 10 client.4564.objecter successfully canceled tick 2023-09-30T00:49:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3128555275 >> v2:172.21.15.135:6800/116426 conn(0x7f7c80068890 msgr2=0x7f7c8006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 1 --2- 172.21.15.40:0/3128555275 >> v2:172.21.15.135:6800/116426 conn(0x7f7c80068890 0x7f7c8006ad40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7c84006fd0 tx=0x7f7c84008040 comp rx=0 tx=0).stop 2023-09-30T00:49:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 10 monclient: shutdown 2023-09-30T00:49:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:53.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3128555275 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c9413ac20 msgr2=0x7f7c940aa040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 1 --2- 172.21.15.40:0/3128555275 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c9413ac20 0x7f7c940aa040 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c009df0 tx=0x7f7c8c00b040 comp rx=0 tx=0).stop 2023-09-30T00:49:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3128555275 shutdown_connections 2023-09-30T00:49:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 1 --2- 172.21.15.40:0/3128555275 >> v2:172.21.15.135:6800/116426 conn(0x7f7c80068890 0x7f7c8006ad40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 1 --2- 172.21.15.40:0/3128555275 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7c9413ac20 0x7f7c940aa040 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 1 --2- 172.21.15.40:0/3128555275 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c940b1460 0x7f7c940a9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 1 --2- 172.21.15.40:0/3128555275 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7c940b1830 0x7f7c9413e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3128555275 >> 172.21.15.40:0/3128555275 conn(0x7f7c940631f0 msgr2=0x7f7c940635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3128555275 shutdown_connections 2023-09-30T00:49:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 1 -- 172.21.15.40:0/3128555275 wait complete. 2023-09-30T00:49:53.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:53.747+0000 7f7c9b24f700 1 librados: shutdown 2023-09-30T00:49:56.762 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.897+0000 7fa45927d700 10 monclient: get_monmap_and_config 2023-09-30T00:49:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.897+0000 7fa45927d700 10 monclient: build_initial_monmap 2023-09-30T00:49:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.897+0000 7fa45927d700 10 monclient: monmap: 2023-09-30T00:49:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:49:56.898949+0000 2023-09-30T00:49:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:49:56.898949+0000 2023-09-30T00:49:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:49:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:49:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:49:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:49:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:49:56.899 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:56.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.898+0000 7fa45927d700 1 Processor -- start 2023-09-30T00:49:56.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.898+0000 7fa45927d700 1 -- start start 2023-09-30T00:49:56.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.898+0000 7fa45927d700 10 monclient: init 2023-09-30T00:49:56.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa45927d700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:56.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa45927d700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-09-30T00:49:56.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa45927d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4540a5710 0x7fa4540a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:56.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa45927d700 10 monclient(hunting): picked mon.noname-b con 0x7fa4540a5710 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:56.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa45927d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4540a7b20 addr v1:172.21.15.135:6789/0 2023-09-30T00:49:56.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa45927d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa45413e8e0 addr v1:172.21.15.40:6789/0 2023-09-30T00:49:56.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa45927d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4540a5ae0 con 0x7fa45413e8e0 2023-09-30T00:49:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa45927d700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4540a7ff0 con 0x7fa4540a7b20 2023-09-30T00:49:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa45927d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa45927d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4540aedc0 con 0x7fa4540a5710 2023-09-30T00:49:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa45927d700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa45927d700 10 monclient(hunting): authenticate will time out at 5458.200357s 2023-09-30T00:49:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa4537fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa45413e8e0 0x7fa45413ae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50550/0 (socket says 172.21.15.40:50550) 2023-09-30T00:49:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa452ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4540a5710 0x7fa4540a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.899+0000 7fa4537fe700 1 -- 172.21.15.40:0/81764425 learned_addr learned my addr 172.21.15.40:0/81764425 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:49:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.900+0000 7fa452ffd700 10 monclient(hunting): get_auth_request con 0x7fa4540a5710 auth_method 0 2023-09-30T00:49:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.900+0000 7fa452ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.900+0000 7fa452ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.900+0000 7fa452ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:56.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.901+0000 7fa452ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:56.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.901+0000 7fa452ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:56.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.901+0000 7fa452ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.903+0000 7fa452ffd700 10 monclient(hunting): handle_auth_done global_id 4460 payload 274 2023-09-30T00:49:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.903+0000 7fa452ffd700 1 -- 172.21.15.40:0/81764425 >> v1:172.21.15.135:6789/0 conn(0x7fa4540a7b20 legacy=0x7fa45413e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.903+0000 7fa452ffd700 1 -- 172.21.15.40:0/81764425 >> v1:172.21.15.40:6789/0 conn(0x7fa45413e8e0 legacy=0x7fa45413ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.903+0000 7fa452ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.903+0000 7fa452ffd700 1 monclient: found mon.noname-b 2023-09-30T00:49:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.903+0000 7fa452ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.903+0000 7fa452ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:49:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.903+0000 7fa452ffd700 1 -- 172.21.15.40:0/81764425 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4540aef00 con 0x7fa4540a5710 2023-09-30T00:49:56.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.905+0000 7fa452ffd700 1 --2- 172.21.15.40:0/81764425 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4540a5710 0x7fa4540a75e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa444009910 tx=0x7fa4440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35fb4a9fb1d42cec server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:56.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa451ffb700 1 -- 172.21.15.40:0/81764425 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44400b860 con 0x7fa4540a5710 2023-09-30T00:49:56.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa451ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:56.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa451ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:56.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa451ffb700 10 monclient: dump: 2023-09-30T00:49:56.911 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:56.911 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:56.911 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:56.911 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:56.911 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:56.911 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:56.911 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:56.911 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:56.911 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:56.911 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:56.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa451ffb700 10 monclient: _finish_auth 0 2023-09-30T00:49:56.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa451ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa451ffb700 1 -- 172.21.15.40:0/81764425 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44400b9c0 con 0x7fa4540a5710 2023-09-30T00:49:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa45927d700 5 monclient: authenticate success, global_id 4460 2023-09-30T00:49:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa45927d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa451ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa45927d700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa451ffb700 1 -- 172.21.15.40:0/81764425 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44400bd60 con 0x7fa4540a5710 2023-09-30T00:49:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa451ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa45927d700 4 set_mon_vals no callback set 2023-09-30T00:49:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa451ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa451ffb700 10 monclient: dump: 2023-09-30T00:49:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:56.912 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:56.913 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:56.913 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:56.913 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:56.913 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:56.913 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:56.913 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:56.913 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa45927d700 10 monclient: shutdown 2023-09-30T00:49:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa45927d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa45927d700 1 -- 172.21.15.40:0/81764425 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4540a5710 msgr2=0x7fa4540a75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa45927d700 1 --2- 172.21.15.40:0/81764425 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4540a5710 0x7fa4540a75e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa444009910 tx=0x7fa4440091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa45927d700 1 -- 172.21.15.40:0/81764425 shutdown_connections 2023-09-30T00:49:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa45927d700 1 --2- 172.21.15.40:0/81764425 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4540a5710 0x7fa4540a75e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa45927d700 1 -- 172.21.15.40:0/81764425 >> 172.21.15.40:0/81764425 conn(0x7fa454069080 msgr2=0x7fa4540631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.910+0000 7fa45927d700 1 -- 172.21.15.40:0/81764425 shutdown_connections 2023-09-30T00:49:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.911+0000 7fa45927d700 1 -- 172.21.15.40:0/81764425 wait complete. 2023-09-30T00:49:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.911+0000 7fa45927d700 10 monclient: build_initial_monmap 2023-09-30T00:49:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.911+0000 7fa45927d700 10 monclient: monmap: 2023-09-30T00:49:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:49:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:49:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:49:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:49:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:49:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:49:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:49:56.918 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 1 librados: starting msgr at 2023-09-30T00:49:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 1 librados: starting objecter 2023-09-30T00:49:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 1 Processor -- start 2023-09-30T00:49:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 1 -- start start 2023-09-30T00:49:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 1 librados: setting wanted keys 2023-09-30T00:49:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 1 librados: calling monclient init 2023-09-30T00:49:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 10 monclient: init 2023-09-30T00:49:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:49:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4540a5710 0x7fa4540ac840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 10 monclient(hunting): picked mon.noname-c con 0x7fa4540a5710 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:49:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4540a7b20 0x7fa4540aafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 10 monclient(hunting): picked mon.noname-b con 0x7fa4540a7b20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:49:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa45413e8e0 0x7fa45414e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 10 monclient(hunting): picked mon.noname-a con 0x7fa45413e8e0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:49:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4540ac3d0 con 0x7fa45413e8e0 2023-09-30T00:49:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4540aa380 con 0x7fa4540a7b20 2023-09-30T00:49:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45414e970 con 0x7fa4540a5710 2023-09-30T00:49:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.912+0000 7fa45927d700 10 monclient(hunting): authenticate will time out at 5458.213866s 2023-09-30T00:49:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa4527fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4540a7b20 0x7fa4540aafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa4537fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa45413e8e0 0x7fa45414e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa452ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4540a5710 0x7fa4540ac840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa452ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4540a5710 0x7fa4540ac840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:58236/0 (socket says 172.21.15.40:58236) 2023-09-30T00:49:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa452ffd700 1 -- 172.21.15.40:0/1111639673 learned_addr learned my addr 172.21.15.40:0/1111639673 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:49:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa452ffd700 10 monclient(hunting): get_auth_request con 0x7fa4540a5710 auth_method 0 2023-09-30T00:49:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa452ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa452ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa452ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa4527fc700 10 monclient(hunting): get_auth_request con 0x7fa4540a7b20 auth_method 0 2023-09-30T00:49:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa4527fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa4527fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:56.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa4527fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa4537fe700 10 monclient(hunting): get_auth_request con 0x7fa45413e8e0 auth_method 0 2023-09-30T00:49:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa4537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa4537fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.913+0000 7fa4537fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.914+0000 7fa452ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.914+0000 7fa452ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.914+0000 7fa452ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.914+0000 7fa4527fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.914+0000 7fa4527fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.914+0000 7fa4527fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.914+0000 7fa4537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:56.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.914+0000 7fa4537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.914+0000 7fa4537fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.917+0000 7fa452ffd700 10 monclient(hunting): handle_auth_done global_id 4463 payload 824 2023-09-30T00:49:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.918+0000 7fa452ffd700 1 -- 172.21.15.40:0/1111639673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4540a7b20 msgr2=0x7fa4540aafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:56.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.918+0000 7fa452ffd700 1 --2- 172.21.15.40:0/1111639673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4540a7b20 0x7fa4540aafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.918+0000 7fa452ffd700 1 -- 172.21.15.40:0/1111639673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa45413e8e0 msgr2=0x7fa45414e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.918+0000 7fa452ffd700 1 --2- 172.21.15.40:0/1111639673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa45413e8e0 0x7fa45414e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.918+0000 7fa452ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.918+0000 7fa452ffd700 1 monclient: found mon.noname-c 2023-09-30T00:49:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.918+0000 7fa452ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.918+0000 7fa452ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:49:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.918+0000 7fa452ffd700 1 -- 172.21.15.40:0/1111639673 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa444009420 con 0x7fa4540a5710 2023-09-30T00:49:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.920+0000 7fa452ffd700 1 --2- 172.21.15.40:0/1111639673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4540a5710 0x7fa4540ac840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa4440093f0 tx=0x7fa444017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.920+0000 7fa43affd700 10 client.?.objecter ms_handle_connect 0x7fa4540a5710 2023-09-30T00:49:56.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.920+0000 7fa43affd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:49:56.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa43affd700 1 -- 172.21.15.40:0/1111639673 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44400bd40 con 0x7fa4540a5710 2023-09-30T00:49:56.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa43affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:56.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa43affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:56.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa43affd700 10 monclient: dump: 2023-09-30T00:49:56.925 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:56.925 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:56.925 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:56.925 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:56.926 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:56.926 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:56.926 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:56.926 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:56.926 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:56.926 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:56.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa43affd700 10 monclient: _finish_auth 0 2023-09-30T00:49:56.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa43affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:56.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa43affd700 1 -- 172.21.15.40:0/1111639673 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa444024440 con 0x7fa4540a5710 2023-09-30T00:49:56.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa43affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:56.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa43affd700 1 -- 172.21.15.40:0/1111639673 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4440247c0 con 0x7fa4540a5710 2023-09-30T00:49:56.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa43affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:56.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa43affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:56.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa43affd700 10 monclient: dump: 2023-09-30T00:49:56.927 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:49:56.927 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:49:56.927 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:56.927 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:49:56.927 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:56.927 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:49:56.927 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:49:56.927 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:49:56.927 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:49:56.927 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:49:56.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa45927d700 5 monclient: authenticate success, global_id 4463 2023-09-30T00:49:56.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa45927d700 10 monclient: _renew_subs 2023-09-30T00:49:56.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa45927d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:56.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa45927d700 1 -- 172.21.15.40:0/1111639673 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45414b8b0 con 0x7fa4540a5710 2023-09-30T00:49:56.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa4517fa700 4 set_mon_vals no callback set 2023-09-30T00:49:56.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa45927d700 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:49:56.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa45927d700 10 monclient: _renew_subs 2023-09-30T00:49:56.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa45927d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:56.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa45927d700 1 -- 172.21.15.40:0/1111639673 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45414bcf0 con 0x7fa4540a5710 2023-09-30T00:49:56.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.924+0000 7fa45927d700 1 librados: init done 2023-09-30T00:49:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.925+0000 7fa45927d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.925+0000 7fa45927d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.925+0000 7fa45927d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:56.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.925+0000 7fa45927d700 1 -- 172.21.15.40:0/1111639673 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa454072b90 con 0x7fa4540a5710 2023-09-30T00:49:56.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.945+0000 7fa43affd700 1 -- 172.21.15.40:0/1111639673 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa44402d440 con 0x7fa4540a5710 2023-09-30T00:49:56.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.945+0000 7fa43affd700 1 --2- 172.21.15.40:0/1111639673 >> v2:172.21.15.135:6800/116426 conn(0x7fa440068890 0x7fa44006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:56.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.946+0000 7fa4527fc700 1 --2- 172.21.15.40:0/1111639673 >> v2:172.21.15.135:6800/116426 conn(0x7fa440068890 0x7fa44006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:56.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.946+0000 7fa4527fc700 10 monclient: get_auth_request con 0x7fa440068890 auth_method 0 2023-09-30T00:49:56.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.946+0000 7fa4527fc700 1 --2- 172.21.15.40:0/1111639673 >> v2:172.21.15.135:6800/116426 conn(0x7fa440068890 0x7fa44006ad40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa4540a9df0 tx=0x7fa448007ec0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:56.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.946+0000 7fa43affd700 10 client.4463.objecter ms_handle_connect 0x7fa440068890 2023-09-30T00:49:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.987+0000 7fa43affd700 1 -- 172.21.15.40:0/1111639673 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa444057a30 con 0x7fa4540a5710 2023-09-30T00:49:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.987+0000 7fa43affd700 10 client.4463.objecter ms_dispatch 0x7fa454001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:49:56.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.987+0000 7fa43affd700 3 client.4463.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:49:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.987+0000 7fa43affd700 3 client.4463.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:49:56.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:56.988+0000 7fa43affd700 20 client.4463.objecter dump_active .. 0 homeless 2023-09-30T00:49:57.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.224+0000 7fa43affd700 1 -- 172.21.15.40:0/1111639673 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa444053670 con 0x7fa4540a5710 2023-09-30T00:49:57.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.224+0000 7fa43affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:57.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.224+0000 7fa43affd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:57.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.478+0000 7fa45927d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:57.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.478+0000 7fa45927d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:57.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.478+0000 7fa45927d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:49:57.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.478+0000 7fa45927d700 1 -- 172.21.15.40:0/1111639673 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa4540b1b00 con 0x7fa4540a5710 2023-09-30T00:49:57.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.484+0000 7fa43affd700 1 -- 172.21.15.40:0/1111639673 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa444015100 con 0x7fa4540a5710 2023-09-30T00:49:57.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.484+0000 7fa43affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:57.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.484+0000 7fa43affd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:57.485 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:49:57.485 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:57.488 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.486+0000 7fa45927d700 10 librados: watch_flush enter 2023-09-30T00:49:57.488 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.486+0000 7fa45927d700 10 librados: watch_flush exit 2023-09-30T00:49:57.488 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 20 client.4463.objecter shutdown clearing up homeless session... 2023-09-30T00:49:57.488 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 10 client.4463.objecter successfully canceled tick 2023-09-30T00:49:57.488 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 1 -- 172.21.15.40:0/1111639673 >> v2:172.21.15.135:6800/116426 conn(0x7fa440068890 msgr2=0x7fa44006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:57.488 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 1 --2- 172.21.15.40:0/1111639673 >> v2:172.21.15.135:6800/116426 conn(0x7fa440068890 0x7fa44006ad40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa4540a9df0 tx=0x7fa448007ec0 comp rx=0 tx=0).stop 2023-09-30T00:49:57.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 10 monclient: shutdown 2023-09-30T00:49:57.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:57.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 1 -- 172.21.15.40:0/1111639673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4540a5710 msgr2=0x7fa4540ac840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:57.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 1 --2- 172.21.15.40:0/1111639673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4540a5710 0x7fa4540ac840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa4440093f0 tx=0x7fa444017000 comp rx=0 tx=0).stop 2023-09-30T00:49:57.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 1 -- 172.21.15.40:0/1111639673 shutdown_connections 2023-09-30T00:49:57.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 1 --2- 172.21.15.40:0/1111639673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa45413e8e0 0x7fa45414e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:57.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 1 --2- 172.21.15.40:0/1111639673 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa4540a5710 0x7fa4540ac840 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:57.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 1 --2- 172.21.15.40:0/1111639673 >> v2:172.21.15.135:6800/116426 conn(0x7fa440068890 0x7fa44006ad40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:57.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 1 --2- 172.21.15.40:0/1111639673 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa4540a7b20 0x7fa4540aafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:57.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 1 -- 172.21.15.40:0/1111639673 >> 172.21.15.40:0/1111639673 conn(0x7fa454069080 msgr2=0x7fa4540b4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:57.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 1 -- 172.21.15.40:0/1111639673 shutdown_connections 2023-09-30T00:49:57.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 1 -- 172.21.15.40:0/1111639673 wait complete. 2023-09-30T00:49:57.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:49:57.487+0000 7fa45927d700 1 librados: shutdown 2023-09-30T00:50:00.011 INFO:tasks.ceph.mon.a.smithi040.stderr:2023-09-30T00:50:00.009+0000 11417700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:50:00.502 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:00.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.638+0000 7fe3d3818700 10 monclient: get_monmap_and_config 2023-09-30T00:50:00.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.638+0000 7fe3d3818700 10 monclient: build_initial_monmap 2023-09-30T00:50:00.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.638+0000 7fe3d3818700 10 monclient: monmap: 2023-09-30T00:50:00.639 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:00.639 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:00.639 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:00.639531+0000 2023-09-30T00:50:00.639 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:00.639531+0000 2023-09-30T00:50:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:00.640 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:00.640 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:00.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.639+0000 7fe3d3818700 1 Processor -- start 2023-09-30T00:50:00.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.639+0000 7fe3d3818700 1 -- start start 2023-09-30T00:50:00.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.639+0000 7fe3d3818700 10 monclient: init 2023-09-30T00:50:00.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.640+0000 7fe3d3818700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:00.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.640+0000 7fe3d3818700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-09-30T00:50:00.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.640+0000 7fe3d3818700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3cc0a98d0 addr v1:172.21.15.135:6789/0 2023-09-30T00:50:00.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.640+0000 7fe3d3818700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3cc0a9ca0 addr v1:172.21.15.40:6789/0 2023-09-30T00:50:00.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.640+0000 7fe3d3818700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe3cc13bc10 0x7fe3cc149c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:00.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.640+0000 7fe3d3818700 10 monclient(hunting): picked mon.noname-c con 0x7fe3cc13bc10 addr v2:172.21.15.135:3300/0 2023-09-30T00:50:00.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.640+0000 7fe3d3818700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3cc06bd60 con 0x7fe3cc0a9ca0 2023-09-30T00:50:00.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.640+0000 7fe3d3818700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3cc06fb90 con 0x7fe3cc0a98d0 2023-09-30T00:50:00.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.640+0000 7fe3d3818700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:00.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.640+0000 7fe3d3818700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3cc06bfd0 con 0x7fe3cc13bc10 2023-09-30T00:50:00.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.640+0000 7fe3d3818700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:00.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.640+0000 7fe3d3818700 10 monclient(hunting): authenticate will time out at 5461.941440s 2023-09-30T00:50:00.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.640+0000 7fe3d0db3700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe3cc0a9ca0 0x7fe3cc13f760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50566/0 (socket says 172.21.15.40:50566) 2023-09-30T00:50:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.640+0000 7fe3d0db3700 1 -- 172.21.15.40:0/4008208859 learned_addr learned my addr 172.21.15.40:0/4008208859 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:50:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.641+0000 7fe3d1db5700 1 --2- 172.21.15.40:0/4008208859 >> v2:172.21.15.135:3300/0 conn(0x7fe3cc13bc10 0x7fe3cc149c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.641+0000 7fe3d1db5700 10 monclient(hunting): get_auth_request con 0x7fe3cc13bc10 auth_method 0 2023-09-30T00:50:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.641+0000 7fe3d1db5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.641+0000 7fe3d1db5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.641+0000 7fe3d1db5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.642+0000 7fe3d1db5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.642+0000 7fe3d1db5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:00.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.642+0000 7fe3d1db5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:00.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.644+0000 7fe3d1db5700 10 monclient(hunting): handle_auth_done global_id 4573 payload 274 2023-09-30T00:50:00.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.644+0000 7fe3d1db5700 1 -- 172.21.15.40:0/4008208859 >> v1:172.21.15.135:6789/0 conn(0x7fe3cc0a98d0 legacy=0x7fe3cc13b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:00.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.644+0000 7fe3c3fff700 1 -- 172.21.15.40:0/4008208859 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913870006 0 0) 0x7fe3cc06bd60 con 0x7fe3cc0a9ca0 2023-09-30T00:50:00.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.644+0000 7fe3d1db5700 1 -- 172.21.15.40:0/4008208859 >> v1:172.21.15.40:6789/0 conn(0x7fe3cc0a9ca0 legacy=0x7fe3cc13f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:00.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.644+0000 7fe3d1db5700 10 monclient: _finish_hunting 0 2023-09-30T00:50:00.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.644+0000 7fe3d1db5700 1 monclient: found mon.noname-c 2023-09-30T00:50:00.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.644+0000 7fe3d1db5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:00.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.644+0000 7fe3d1db5700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:50:00.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.644+0000 7fe3d1db5700 1 -- 172.21.15.40:0/4008208859 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc06c2b0 con 0x7fe3cc13bc10 2023-09-30T00:50:00.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.644+0000 7fe3c3fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T00:50:00.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.646+0000 7fe3d1db5700 1 --2- 172.21.15.40:0/4008208859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3cc13bc10 0x7fe3cc149c10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4009e60 tx=0x7fe3c4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c91b4d931aaa6dd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3c3fff700 1 -- 172.21.15.40:0/4008208859 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c400e070 con 0x7fe3cc13bc10 2023-09-30T00:50:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3c3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:50:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3c3fff700 10 monclient: dump: 2023-09-30T00:50:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:00.653 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3c3fff700 10 monclient: _finish_auth 0 2023-09-30T00:50:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3c3fff700 1 -- 172.21.15.40:0/4008208859 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c400bbf0 con 0x7fe3cc13bc10 2023-09-30T00:50:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3c3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3c3fff700 1 -- 172.21.15.40:0/4008208859 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4012430 con 0x7fe3cc13bc10 2023-09-30T00:50:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3c3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:50:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3c3fff700 10 monclient: dump: 2023-09-30T00:50:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:00.654 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3d3818700 5 monclient: authenticate success, global_id 4573 2023-09-30T00:50:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3d3818700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.651+0000 7fe3d3818700 4 set_mon_vals no callback set 2023-09-30T00:50:00.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.652+0000 7fe3d3818700 10 monclient: shutdown 2023-09-30T00:50:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.652+0000 7fe3d3818700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.652+0000 7fe3d3818700 1 -- 172.21.15.40:0/4008208859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3cc13bc10 msgr2=0x7fe3cc149c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.652+0000 7fe3d3818700 1 --2- 172.21.15.40:0/4008208859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3cc13bc10 0x7fe3cc149c10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4009e60 tx=0x7fe3c4009780 comp rx=0 tx=0).stop 2023-09-30T00:50:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.652+0000 7fe3d3818700 1 -- 172.21.15.40:0/4008208859 shutdown_connections 2023-09-30T00:50:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.652+0000 7fe3d3818700 1 --2- 172.21.15.40:0/4008208859 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3cc13bc10 0x7fe3cc149c10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.652+0000 7fe3d3818700 1 -- 172.21.15.40:0/4008208859 >> 172.21.15.40:0/4008208859 conn(0x7fe3cc0680a0 msgr2=0x7fe3cc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.652+0000 7fe3d3818700 1 -- 172.21.15.40:0/4008208859 shutdown_connections 2023-09-30T00:50:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.652+0000 7fe3d3818700 1 -- 172.21.15.40:0/4008208859 wait complete. 2023-09-30T00:50:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.653+0000 7fe3d3818700 10 monclient: build_initial_monmap 2023-09-30T00:50:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.653+0000 7fe3d3818700 10 monclient: monmap: 2023-09-30T00:50:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:00.659 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.653+0000 7fe3d3818700 1 librados: starting msgr at 2023-09-30T00:50:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.653+0000 7fe3d3818700 1 librados: starting objecter 2023-09-30T00:50:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.654+0000 7fe3d3818700 1 Processor -- start 2023-09-30T00:50:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.654+0000 7fe3d3818700 1 -- start start 2023-09-30T00:50:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.654+0000 7fe3d3818700 1 librados: setting wanted keys 2023-09-30T00:50:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.654+0000 7fe3d3818700 1 librados: calling monclient init 2023-09-30T00:50:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.654+0000 7fe3d3818700 10 monclient: init 2023-09-30T00:50:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:50:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3cc0a98d0 0x7fe3cc0ada80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 10 monclient(hunting): picked mon.noname-b con 0x7fe3cc0a98d0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3cc0a9ca0 0x7fe3cc0aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 10 monclient(hunting): picked mon.noname-a con 0x7fe3cc0a9ca0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3cc13bc10 0x7fe3cc0ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 10 monclient(hunting): picked mon.noname-c con 0x7fe3cc13bc10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:00.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:00.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc119dd0 con 0x7fe3cc0a9ca0 2023-09-30T00:50:00.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:00.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc11a040 con 0x7fe3cc0a98d0 2023-09-30T00:50:00.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:00.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc1182d0 con 0x7fe3cc13bc10 2023-09-30T00:50:00.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:00.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.655+0000 7fe3d3818700 10 monclient(hunting): authenticate will time out at 5461.956626s 2023-09-30T00:50:00.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d0db3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3cc0a9ca0 0x7fe3cc0aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:00.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d1db5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3cc13bc10 0x7fe3cc0ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:00.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d15b4700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3cc0a98d0 0x7fe3cc0ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:00.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d0db3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3cc0a9ca0 0x7fe3cc0aaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41608/0 (socket says 172.21.15.40:41608) 2023-09-30T00:50:00.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d0db3700 1 -- 172.21.15.40:0/1661354101 learned_addr learned my addr 172.21.15.40:0/1661354101 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:00.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d0db3700 10 monclient(hunting): get_auth_request con 0x7fe3cc0a9ca0 auth_method 0 2023-09-30T00:50:00.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d0db3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:00.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d0db3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:00.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d0db3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:00.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d15b4700 10 monclient(hunting): get_auth_request con 0x7fe3cc0a98d0 auth_method 0 2023-09-30T00:50:00.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d15b4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:00.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d15b4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:00.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d15b4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:00.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d1db5700 10 monclient(hunting): get_auth_request con 0x7fe3cc13bc10 auth_method 0 2023-09-30T00:50:00.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d1db5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:00.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d1db5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:00.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.656+0000 7fe3d1db5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.657+0000 7fe3d15b4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.657+0000 7fe3d15b4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.657+0000 7fe3d15b4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.657+0000 7fe3d0db3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.657+0000 7fe3d0db3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.657+0000 7fe3d0db3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.657+0000 7fe3d1db5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.657+0000 7fe3d1db5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.657+0000 7fe3d1db5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.660+0000 7fe3d15b4700 10 monclient(hunting): handle_auth_done global_id 4576 payload 824 2023-09-30T00:50:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.660+0000 7fe3d15b4700 1 -- 172.21.15.40:0/1661354101 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3cc13bc10 msgr2=0x7fe3cc0ab430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.660+0000 7fe3d15b4700 1 --2- 172.21.15.40:0/1661354101 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3cc13bc10 0x7fe3cc0ab430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.660+0000 7fe3d15b4700 1 -- 172.21.15.40:0/1661354101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3cc0a9ca0 msgr2=0x7fe3cc0aaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.660+0000 7fe3d15b4700 1 --2- 172.21.15.40:0/1661354101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3cc0a9ca0 0x7fe3cc0aaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.660+0000 7fe3d15b4700 10 monclient: _finish_hunting 0 2023-09-30T00:50:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.660+0000 7fe3d15b4700 1 monclient: found mon.noname-b 2023-09-30T00:50:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.660+0000 7fe3d15b4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.660+0000 7fe3d15b4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:50:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.660+0000 7fe3d15b4700 1 -- 172.21.15.40:0/1661354101 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c4009a20 con 0x7fe3cc0a98d0 2023-09-30T00:50:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.662+0000 7fe3d15b4700 1 --2- 172.21.15.40:0/1661354101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3cc0a98d0 0x7fe3cc0ada80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe3c800b5b0 tx=0x7fe3c80090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.662+0000 7fe3c17fa700 10 client.?.objecter ms_handle_connect 0x7fe3cc0a98d0 2023-09-30T00:50:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.662+0000 7fe3c17fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:00.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3c17fa700 1 -- 172.21.15.40:0/1661354101 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c800f510 con 0x7fe3cc0a98d0 2023-09-30T00:50:00.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:00.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3c17fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:00.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3c17fa700 10 monclient: dump: 2023-09-30T00:50:00.667 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:00.668 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3c17fa700 10 monclient: _finish_auth 0 2023-09-30T00:50:00.668 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3c17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3d3818700 5 monclient: authenticate success, global_id 4576 2023-09-30T00:50:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3c17fa700 1 -- 172.21.15.40:0/1661354101 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c800f690 con 0x7fe3cc0a98d0 2023-09-30T00:50:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3c17fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3d3818700 10 monclient: _renew_subs 2023-09-30T00:50:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3c17fa700 1 -- 172.21.15.40:0/1661354101 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c800fa50 con 0x7fe3cc0a98d0 2023-09-30T00:50:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3d3818700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3d3818700 1 -- 172.21.15.40:0/1661354101 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3cc1185b0 con 0x7fe3cc0a98d0 2023-09-30T00:50:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3c37fe700 4 set_mon_vals no callback set 2023-09-30T00:50:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3c17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:50:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3c17fa700 10 monclient: dump: 2023-09-30T00:50:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:00.670 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3d3818700 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3d3818700 10 monclient: _renew_subs 2023-09-30T00:50:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3d3818700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:00.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3d3818700 1 -- 172.21.15.40:0/1661354101 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3cc118ec0 con 0x7fe3cc0a98d0 2023-09-30T00:50:00.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.666+0000 7fe3d3818700 1 librados: init done 2023-09-30T00:50:00.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.667+0000 7fe3d3818700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:00.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.667+0000 7fe3d3818700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:00.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.667+0000 7fe3d3818700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:00.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.667+0000 7fe3d3818700 1 -- 172.21.15.40:0/1661354101 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3cc1187f0 con 0x7fe3cc0a98d0 2023-09-30T00:50:00.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.685+0000 7fe3c17fa700 1 -- 172.21.15.40:0/1661354101 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fe3c8024430 con 0x7fe3cc0a98d0 2023-09-30T00:50:00.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.686+0000 7fe3c17fa700 1 --2- 172.21.15.40:0/1661354101 >> v2:172.21.15.135:6800/116426 conn(0x7fe3b0068960 0x7fe3b006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:00.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.686+0000 7fe3d0db3700 1 --2- 172.21.15.40:0/1661354101 >> v2:172.21.15.135:6800/116426 conn(0x7fe3b0068960 0x7fe3b006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:00.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.686+0000 7fe3d0db3700 10 monclient: get_auth_request con 0x7fe3b0068960 auth_method 0 2023-09-30T00:50:00.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.686+0000 7fe3c17fa700 10 client.4576.objecter ms_handle_connect 0x7fe3b0068960 2023-09-30T00:50:00.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.686+0000 7fe3d0db3700 1 --2- 172.21.15.40:0/1661354101 >> v2:172.21.15.135:6800/116426 conn(0x7fe3b0068960 0x7fe3b006ae10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc13dae0 tx=0x7fe3bc007df0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:00.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.916+0000 7fe3c17fa700 1 -- 172.21.15.40:0/1661354101 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fe3c8055f60 con 0x7fe3cc0a98d0 2023-09-30T00:50:00.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.916+0000 7fe3c17fa700 10 client.4576.objecter ms_dispatch 0x7fe3cc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:00.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.917+0000 7fe3c17fa700 3 client.4576.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:00.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.917+0000 7fe3c17fa700 3 client.4576.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.917+0000 7fe3c17fa700 20 client.4576.objecter dump_active .. 0 homeless 2023-09-30T00:50:00.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.935+0000 7fe3c17fa700 1 -- 172.21.15.40:0/1661354101 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe3c8051ba0 con 0x7fe3cc0a98d0 2023-09-30T00:50:00.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.935+0000 7fe3c17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:00.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:00.935+0000 7fe3c17fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.190+0000 7fe3d3818700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.190+0000 7fe3d3818700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.190+0000 7fe3d3818700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:01.191 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.190+0000 7fe3d3818700 1 -- 172.21.15.40:0/1661354101 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe3cc072b90 con 0x7fe3cc0a98d0 2023-09-30T00:50:01.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.195+0000 7fe3c17fa700 1 -- 172.21.15.40:0/1661354101 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe3c80551c0 con 0x7fe3cc0a98d0 2023-09-30T00:50:01.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.195+0000 7fe3c17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:01.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.195+0000 7fe3c17fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:01.196 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:01.196 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:01.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 10 librados: watch_flush enter 2023-09-30T00:50:01.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 10 librados: watch_flush exit 2023-09-30T00:50:01.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 20 client.4576.objecter shutdown clearing up homeless session... 2023-09-30T00:50:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 10 client.4576.objecter successfully canceled tick 2023-09-30T00:50:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 1 -- 172.21.15.40:0/1661354101 >> v2:172.21.15.135:6800/116426 conn(0x7fe3b0068960 msgr2=0x7fe3b006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 1 --2- 172.21.15.40:0/1661354101 >> v2:172.21.15.135:6800/116426 conn(0x7fe3b0068960 0x7fe3b006ae10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc13dae0 tx=0x7fe3bc007df0 comp rx=0 tx=0).stop 2023-09-30T00:50:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 10 monclient: shutdown 2023-09-30T00:50:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 1 -- 172.21.15.40:0/1661354101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3cc0a98d0 msgr2=0x7fe3cc0ada80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 1 --2- 172.21.15.40:0/1661354101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3cc0a98d0 0x7fe3cc0ada80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe3c800b5b0 tx=0x7fe3c80090f0 comp rx=0 tx=0).stop 2023-09-30T00:50:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 1 -- 172.21.15.40:0/1661354101 shutdown_connections 2023-09-30T00:50:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 1 --2- 172.21.15.40:0/1661354101 >> v2:172.21.15.135:6800/116426 conn(0x7fe3b0068960 0x7fe3b006ae10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 1 --2- 172.21.15.40:0/1661354101 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe3cc0a98d0 0x7fe3cc0ada80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 1 --2- 172.21.15.40:0/1661354101 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe3cc13bc10 0x7fe3cc0ab430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 1 --2- 172.21.15.40:0/1661354101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3cc0a9ca0 0x7fe3cc0aaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 1 -- 172.21.15.40:0/1661354101 >> 172.21.15.40:0/1661354101 conn(0x7fe3cc0680a0 msgr2=0x7fe3cc06da60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 1 -- 172.21.15.40:0/1661354101 shutdown_connections 2023-09-30T00:50:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 1 -- 172.21.15.40:0/1661354101 wait complete. 2023-09-30T00:50:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:01.197+0000 7fe3d3818700 1 librados: shutdown 2023-09-30T00:50:04.213 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:04.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.349+0000 7efc7bde9700 10 monclient: get_monmap_and_config 2023-09-30T00:50:04.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.349+0000 7efc7bde9700 10 monclient: build_initial_monmap 2023-09-30T00:50:04.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.349+0000 7efc7bde9700 10 monclient: monmap: 2023-09-30T00:50:04.350 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:04.351 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:04.351 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:04.350858+0000 2023-09-30T00:50:04.351 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:04.350858+0000 2023-09-30T00:50:04.351 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:04.351 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:04.351 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:04.351 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:04.351 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:04.351 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:04.351 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:04.352 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:04.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.350+0000 7efc7bde9700 1 Processor -- start 2023-09-30T00:50:04.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.350+0000 7efc7bde9700 1 -- start start 2023-09-30T00:50:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.350+0000 7efc7bde9700 10 monclient: init 2023-09-30T00:50:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.351+0000 7efc7bde9700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.351+0000 7efc7bde9700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-09-30T00:50:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.351+0000 7efc7bde9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efc74146060 addr v1:172.21.15.40:6789/0 2023-09-30T00:50:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.351+0000 7efc7bde9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc74146430 0x7efc7406e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.351+0000 7efc7bde9700 10 monclient(hunting): picked mon.noname-b con 0x7efc74146430 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.351+0000 7efc7bde9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc74142d00 addr v1:172.21.15.135:6789/0 2023-09-30T00:50:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.351+0000 7efc7bde9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc74146960 con 0x7efc74146060 2023-09-30T00:50:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.351+0000 7efc7bde9700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7406eb70 con 0x7efc74142d00 2023-09-30T00:50:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.351+0000 7efc7bde9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.351+0000 7efc7bde9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7406d260 con 0x7efc74146430 2023-09-30T00:50:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.351+0000 7efc7bde9700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.351+0000 7efc7bde9700 10 monclient(hunting): authenticate will time out at 5465.652472s 2023-09-30T00:50:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.351+0000 7efc79b85700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efc74146060 0x7efc7406df20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34694/0 (socket says 172.21.15.40:34694) 2023-09-30T00:50:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.352+0000 7efc79b85700 1 -- 172.21.15.40:0/174781480 learned_addr learned my addr 172.21.15.40:0/174781480 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:50:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.352+0000 7efc79384700 1 --2- 172.21.15.40:0/174781480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc74146430 0x7efc7406e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.352+0000 7efc79384700 10 monclient(hunting): get_auth_request con 0x7efc74146430 auth_method 0 2023-09-30T00:50:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.352+0000 7efc79384700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.352+0000 7efc79384700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.352+0000 7efc79384700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.353+0000 7efc79384700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.353+0000 7efc79384700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.353+0000 7efc79384700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc78b83700 1 -- 172.21.15.40:0/174781480 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839250372 0 0) 0x7efc7406eb70 con 0x7efc74142d00 2023-09-30T00:50:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc78b83700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc78b83700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc78b83700 10 monclient(hunting): my global_id is 4579 2023-09-30T00:50:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc78b83700 1 -- 172.21.15.40:0/174781480 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc60003570 con 0x7efc74142d00 2023-09-30T00:50:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc78b83700 1 -- 172.21.15.40:0/174781480 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837901580 0 0) 0x7efc74146960 con 0x7efc74146060 2023-09-30T00:50:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc78b83700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:04.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc78b83700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc78b83700 10 monclient(hunting): my global_id is 4524 2023-09-30T00:50:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc78b83700 1 -- 172.21.15.40:0/174781480 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc7406eb70 con 0x7efc74146060 2023-09-30T00:50:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc79384700 10 monclient(hunting): handle_auth_done global_id 4469 payload 274 2023-09-30T00:50:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc79384700 1 -- 172.21.15.40:0/174781480 >> v1:172.21.15.135:6789/0 conn(0x7efc74142d00 legacy=0x7efc74143170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc79384700 1 -- 172.21.15.40:0/174781480 >> v1:172.21.15.40:6789/0 conn(0x7efc74146060 legacy=0x7efc7406df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc79384700 10 monclient: _finish_hunting 0 2023-09-30T00:50:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.355+0000 7efc79384700 1 monclient: found mon.noname-b 2023-09-30T00:50:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.356+0000 7efc79384700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.356+0000 7efc79384700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:50:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.356+0000 7efc79384700 1 -- 172.21.15.40:0/174781480 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7406d540 con 0x7efc74146430 2023-09-30T00:50:04.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.358+0000 7efc79384700 1 --2- 172.21.15.40:0/174781480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc74146430 0x7efc7406e630 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efc64009910 tx=0x7efc640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c48d450e1b2f7a26 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc78b83700 1 -- 172.21.15.40:0/174781480 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64011630 con 0x7efc74146430 2023-09-30T00:50:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc78b83700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc78b83700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc78b83700 10 monclient: dump: 2023-09-30T00:50:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:04.364 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc78b83700 10 monclient: _finish_auth 0 2023-09-30T00:50:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc78b83700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc78b83700 1 -- 172.21.15.40:0/174781480 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc64011790 con 0x7efc74146430 2023-09-30T00:50:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc78b83700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc78b83700 1 -- 172.21.15.40:0/174781480 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64011b30 con 0x7efc74146430 2023-09-30T00:50:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc7bde9700 5 monclient: authenticate success, global_id 4469 2023-09-30T00:50:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc7bde9700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc78b83700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc7bde9700 4 set_mon_vals no callback set 2023-09-30T00:50:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc78b83700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:50:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.362+0000 7efc78b83700 10 monclient: dump: 2023-09-30T00:50:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:04.366 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.363+0000 7efc7bde9700 10 monclient: shutdown 2023-09-30T00:50:04.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.363+0000 7efc7bde9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:04.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.363+0000 7efc7bde9700 1 -- 172.21.15.40:0/174781480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc74146430 msgr2=0x7efc7406e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:04.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.363+0000 7efc7bde9700 1 --2- 172.21.15.40:0/174781480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc74146430 0x7efc7406e630 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efc64009910 tx=0x7efc640091a0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.363+0000 7efc7bde9700 1 -- 172.21.15.40:0/174781480 shutdown_connections 2023-09-30T00:50:04.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.363+0000 7efc7bde9700 1 --2- 172.21.15.40:0/174781480 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc74146430 0x7efc7406e630 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.363+0000 7efc7bde9700 1 -- 172.21.15.40:0/174781480 >> 172.21.15.40:0/174781480 conn(0x7efc740680a0 msgr2=0x7efc740b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:04.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.363+0000 7efc7bde9700 1 -- 172.21.15.40:0/174781480 shutdown_connections 2023-09-30T00:50:04.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.363+0000 7efc7bde9700 1 -- 172.21.15.40:0/174781480 wait complete. 2023-09-30T00:50:04.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.364+0000 7efc7bde9700 10 monclient: build_initial_monmap 2023-09-30T00:50:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.364+0000 7efc7bde9700 10 monclient: monmap: 2023-09-30T00:50:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:04.372 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:04.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.364+0000 7efc7bde9700 1 librados: starting msgr at 2023-09-30T00:50:04.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.364+0000 7efc7bde9700 1 librados: starting objecter 2023-09-30T00:50:04.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.364+0000 7efc7bde9700 1 Processor -- start 2023-09-30T00:50:04.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.364+0000 7efc7bde9700 1 -- start start 2023-09-30T00:50:04.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.364+0000 7efc7bde9700 1 librados: setting wanted keys 2023-09-30T00:50:04.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.364+0000 7efc7bde9700 1 librados: calling monclient init 2023-09-30T00:50:04.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.364+0000 7efc7bde9700 10 monclient: init 2023-09-30T00:50:04.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:04.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:50:04.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc74142d00 0x7efc740af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:04.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 10 monclient(hunting): picked mon.noname-c con 0x7efc74142d00 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:04.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc74146060 0x7efc74148720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:04.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 10 monclient(hunting): picked mon.noname-a con 0x7efc74146060 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:04.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc74146430 0x7efc741494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:04.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 10 monclient(hunting): picked mon.noname-b con 0x7efc74146430 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:04.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:04.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc740a62c0 con 0x7efc74146060 2023-09-30T00:50:04.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:04.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc74148c60 con 0x7efc74146430 2023-09-30T00:50:04.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:04.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc740b0410 con 0x7efc74142d00 2023-09-30T00:50:04.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:04.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.365+0000 7efc7bde9700 10 monclient(hunting): authenticate will time out at 5465.666569s 2023-09-30T00:50:04.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.366+0000 7efc79384700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc74146060 0x7efc74148720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:04.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.366+0000 7efc7a386700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc74146430 0x7efc741494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:04.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.366+0000 7efc7a386700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc74146430 0x7efc741494a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:57312/0 (socket says 172.21.15.40:57312) 2023-09-30T00:50:04.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.366+0000 7efc7a386700 1 -- 172.21.15.40:0/3765654320 learned_addr learned my addr 172.21.15.40:0/3765654320 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:04.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.366+0000 7efc79384700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc74146060 0x7efc74148720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56662/0 (socket says 172.21.15.40:56662) 2023-09-30T00:50:04.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.366+0000 7efc7a386700 10 monclient(hunting): get_auth_request con 0x7efc74146430 auth_method 0 2023-09-30T00:50:04.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.366+0000 7efc7a386700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:04.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.366+0000 7efc7a386700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:04.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.366+0000 7efc7a386700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:04.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.366+0000 7efc79384700 10 monclient(hunting): get_auth_request con 0x7efc74146060 auth_method 0 2023-09-30T00:50:04.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.366+0000 7efc79384700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:04.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.366+0000 7efc79384700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:04.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.366+0000 7efc79384700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:04.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.367+0000 7efc7a386700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:04.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.367+0000 7efc7a386700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:04.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.367+0000 7efc7a386700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:04.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.367+0000 7efc79384700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:04.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.367+0000 7efc79384700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:04.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.367+0000 7efc79384700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:04.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.367+0000 7efc79b85700 1 --2- 172.21.15.40:0/3765654320 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc74142d00 0x7efc740af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:04.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.367+0000 7efc79b85700 10 monclient(hunting): get_auth_request con 0x7efc74142d00 auth_method 0 2023-09-30T00:50:04.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.367+0000 7efc79b85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:04.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.367+0000 7efc79b85700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:04.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.367+0000 7efc79b85700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:04.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.368+0000 7efc79b85700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:04.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.368+0000 7efc79b85700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:04.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.368+0000 7efc79b85700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:04.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.370+0000 7efc7a386700 10 monclient(hunting): handle_auth_done global_id 4582 payload 824 2023-09-30T00:50:04.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.370+0000 7efc7a386700 1 -- 172.21.15.40:0/3765654320 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc74142d00 msgr2=0x7efc740af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.370+0000 7efc7a386700 1 --2- 172.21.15.40:0/3765654320 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc74142d00 0x7efc740af7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.370+0000 7efc7a386700 1 -- 172.21.15.40:0/3765654320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc74146060 msgr2=0x7efc74148720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.370+0000 7efc7a386700 1 --2- 172.21.15.40:0/3765654320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc74146060 0x7efc74148720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.370+0000 7efc7a386700 10 monclient: _finish_hunting 0 2023-09-30T00:50:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.370+0000 7efc7a386700 1 monclient: found mon.noname-b 2023-09-30T00:50:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.370+0000 7efc7a386700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.370+0000 7efc7a386700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:50:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.370+0000 7efc7a386700 1 -- 172.21.15.40:0/3765654320 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc64009420 con 0x7efc74146430 2023-09-30T00:50:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.373+0000 7efc7a386700 1 --2- 172.21.15.40:0/3765654320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc74146430 0x7efc741494a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00c910 tx=0x7efc6c0027a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:04.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.373+0000 7efc69ffb700 10 client.?.objecter ms_handle_connect 0x7efc74146430 2023-09-30T00:50:04.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.373+0000 7efc69ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:04.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.377+0000 7efc69ffb700 1 -- 172.21.15.40:0/3765654320 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c012630 con 0x7efc74146430 2023-09-30T00:50:04.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.377+0000 7efc69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:04.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.377+0000 7efc69ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:04.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.377+0000 7efc69ffb700 10 monclient: dump: 2023-09-30T00:50:04.380 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:04.380 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:04.381 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.377+0000 7efc69ffb700 10 monclient: _finish_auth 0 2023-09-30T00:50:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.377+0000 7efc69ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.377+0000 7efc69ffb700 1 -- 172.21.15.40:0/3765654320 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c0127b0 con 0x7efc74146430 2023-09-30T00:50:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.377+0000 7efc69ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.377+0000 7efc69ffb700 1 -- 172.21.15.40:0/3765654320 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c012b70 con 0x7efc74146430 2023-09-30T00:50:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.377+0000 7efc7bde9700 5 monclient: authenticate success, global_id 4582 2023-09-30T00:50:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.377+0000 7efc7bde9700 10 monclient: _renew_subs 2023-09-30T00:50:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.377+0000 7efc7bde9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.377+0000 7efc7bde9700 1 -- 172.21.15.40:0/3765654320 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc741499e0 con 0x7efc74146430 2023-09-30T00:50:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.377+0000 7efc6bfff700 4 set_mon_vals no callback set 2023-09-30T00:50:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.378+0000 7efc69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.378+0000 7efc69ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:50:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.378+0000 7efc69ffb700 10 monclient: dump: 2023-09-30T00:50:04.383 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:04.383 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:04.383 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:04.383 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:04.383 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:04.383 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:04.383 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:04.383 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:04.383 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:04.383 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:04.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.378+0000 7efc7bde9700 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:04.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.378+0000 7efc7bde9700 10 monclient: _renew_subs 2023-09-30T00:50:04.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.378+0000 7efc7bde9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:04.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.378+0000 7efc7bde9700 1 -- 172.21.15.40:0/3765654320 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc74147160 con 0x7efc74146430 2023-09-30T00:50:04.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.378+0000 7efc7bde9700 1 librados: init done 2023-09-30T00:50:04.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.379+0000 7efc7bde9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:04.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.379+0000 7efc7bde9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:04.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.379+0000 7efc7bde9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:04.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.379+0000 7efc7bde9700 1 -- 172.21.15.40:0/3765654320 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc74072b90 con 0x7efc74146430 2023-09-30T00:50:04.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.398+0000 7efc69ffb700 1 -- 172.21.15.40:0/3765654320 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7efc6c01b400 con 0x7efc74146430 2023-09-30T00:50:04.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.398+0000 7efc69ffb700 1 --2- 172.21.15.40:0/3765654320 >> v2:172.21.15.135:6800/116426 conn(0x7efc58068960 0x7efc5806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:04.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.399+0000 7efc79b85700 1 --2- 172.21.15.40:0/3765654320 >> v2:172.21.15.135:6800/116426 conn(0x7efc58068960 0x7efc5806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:04.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.399+0000 7efc79b85700 10 monclient: get_auth_request con 0x7efc58068960 auth_method 0 2023-09-30T00:50:04.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.399+0000 7efc79b85700 1 --2- 172.21.15.40:0/3765654320 >> v2:172.21.15.135:6800/116426 conn(0x7efc58068960 0x7efc5806ae10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efc70007fd0 tx=0x7efc70009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:04.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.399+0000 7efc69ffb700 10 client.4582.objecter ms_handle_connect 0x7efc58068960 2023-09-30T00:50:04.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.531+0000 7efc69ffb700 1 -- 172.21.15.40:0/3765654320 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7efc6c056440 con 0x7efc74146430 2023-09-30T00:50:04.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.531+0000 7efc69ffb700 10 client.4582.objecter ms_dispatch 0x7efc74001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:04.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.531+0000 7efc69ffb700 3 client.4582.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:04.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.531+0000 7efc69ffb700 3 client.4582.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:04.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.531+0000 7efc69ffb700 20 client.4582.objecter dump_active .. 0 homeless 2023-09-30T00:50:04.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.639+0000 7efc69ffb700 1 -- 172.21.15.40:0/3765654320 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efc6c052080 con 0x7efc74146430 2023-09-30T00:50:04.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.639+0000 7efc69ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:04.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.639+0000 7efc69ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.896+0000 7efc7bde9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.896+0000 7efc7bde9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.896+0000 7efc7bde9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:04.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.896+0000 7efc7bde9700 1 -- 172.21.15.40:0/3765654320 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efc7406d820 con 0x7efc74146430 2023-09-30T00:50:04.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.902+0000 7efc69ffb700 1 -- 172.21.15.40:0/3765654320 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7efc6c016100 con 0x7efc74146430 2023-09-30T00:50:04.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.902+0000 7efc69ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:04.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.902+0000 7efc69ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:04.903 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:04.903 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:04.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 10 librados: watch_flush enter 2023-09-30T00:50:04.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 10 librados: watch_flush exit 2023-09-30T00:50:04.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 20 client.4582.objecter shutdown clearing up homeless session... 2023-09-30T00:50:04.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 10 client.4582.objecter successfully canceled tick 2023-09-30T00:50:04.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 1 -- 172.21.15.40:0/3765654320 >> v2:172.21.15.135:6800/116426 conn(0x7efc58068960 msgr2=0x7efc5806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:04.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 1 --2- 172.21.15.40:0/3765654320 >> v2:172.21.15.135:6800/116426 conn(0x7efc58068960 0x7efc5806ae10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efc70007fd0 tx=0x7efc70009040 comp rx=0 tx=0).stop 2023-09-30T00:50:04.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 10 monclient: shutdown 2023-09-30T00:50:04.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 1 -- 172.21.15.40:0/3765654320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc74146430 msgr2=0x7efc741494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 1 --2- 172.21.15.40:0/3765654320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc74146430 0x7efc741494a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00c910 tx=0x7efc6c0027a0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 1 -- 172.21.15.40:0/3765654320 shutdown_connections 2023-09-30T00:50:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 1 --2- 172.21.15.40:0/3765654320 >> v2:172.21.15.135:6800/116426 conn(0x7efc58068960 0x7efc5806ae10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 1 --2- 172.21.15.40:0/3765654320 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc74146430 0x7efc741494a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 1 --2- 172.21.15.40:0/3765654320 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc74142d00 0x7efc740af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 1 --2- 172.21.15.40:0/3765654320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc74146060 0x7efc74148720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 1 -- 172.21.15.40:0/3765654320 >> 172.21.15.40:0/3765654320 conn(0x7efc740680a0 msgr2=0x7efc74141e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 1 -- 172.21.15.40:0/3765654320 shutdown_connections 2023-09-30T00:50:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 1 -- 172.21.15.40:0/3765654320 wait complete. 2023-09-30T00:50:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:04.904+0000 7efc7bde9700 1 librados: shutdown 2023-09-30T00:50:07.919 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:08.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.057+0000 7f768f760700 10 monclient: get_monmap_and_config 2023-09-30T00:50:08.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.057+0000 7f768f760700 10 monclient: build_initial_monmap 2023-09-30T00:50:08.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.057+0000 7f768f760700 10 monclient: monmap: 2023-09-30T00:50:08.058 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:08.058 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:08.058 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:08.058757+0000 2023-09-30T00:50:08.059 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:08.058757+0000 2023-09-30T00:50:08.059 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:08.059 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:08.059 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:08.059 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:08.059 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:08.059 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:08.059 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:08.059 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:08.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.058+0000 7f768f760700 1 Processor -- start 2023-09-30T00:50:08.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.058+0000 7f768f760700 1 -- start start 2023-09-30T00:50:08.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.058+0000 7f768f760700 10 monclient: init 2023-09-30T00:50:08.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.059+0000 7f768f760700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:08.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.059+0000 7f768f760700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2023-09-30T00:50:08.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.059+0000 7f768f760700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f76880b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:50:08.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.059+0000 7f768f760700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f76880b1830 0x7f76880a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:08.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.059+0000 7f768f760700 10 monclient(hunting): picked mon.noname-a con 0x7f76880b1830 addr v2:172.21.15.40:3300/0 2023-09-30T00:50:08.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.059+0000 7f768f760700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f768813ac20 addr v1:172.21.15.135:6789/0 2023-09-30T00:50:08.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.059+0000 7f768f760700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76880a5b20 con 0x7f76880b1460 2023-09-30T00:50:08.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.059+0000 7f768f760700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76880a71f0 con 0x7f768813ac20 2023-09-30T00:50:08.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.059+0000 7f768f760700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:08.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.059+0000 7f768f760700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7688141e40 con 0x7f76880b1830 2023-09-30T00:50:08.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.059+0000 7f768f760700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:08.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.059+0000 7f768f760700 10 monclient(hunting): authenticate will time out at 5469.360416s 2023-09-30T00:50:08.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.060+0000 7f768ccfb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f76880b1830 0x7f76880a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:08.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.060+0000 7f768ccfb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f76880b1830 0x7f76880a5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56676/0 (socket says 172.21.15.40:56676) 2023-09-30T00:50:08.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.060+0000 7f768ccfb700 1 -- 172.21.15.40:0/990043595 learned_addr learned my addr 172.21.15.40:0/990043595 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:08.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.060+0000 7f768ccfb700 10 monclient(hunting): get_auth_request con 0x7f76880b1830 auth_method 0 2023-09-30T00:50:08.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.060+0000 7f768ccfb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:08.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.060+0000 7f768ccfb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:08.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.060+0000 7f768ccfb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:08.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.061+0000 7f768ccfb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:08.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.061+0000 7f768ccfb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:08.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.061+0000 7f768ccfb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:08.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.064+0000 7f768ccfb700 10 monclient(hunting): handle_auth_done global_id 4530 payload 274 2023-09-30T00:50:08.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.064+0000 7f768ccfb700 1 -- 172.21.15.40:0/990043595 >> v1:172.21.15.135:6789/0 conn(0x7f768813ac20 legacy=0x7f768813b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:08.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.064+0000 7f768ccfb700 1 -- 172.21.15.40:0/990043595 >> v1:172.21.15.40:6789/0 conn(0x7f76880b1460 legacy=0x7f7688141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:08.065 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.064+0000 7f768ccfb700 10 monclient: _finish_hunting 0 2023-09-30T00:50:08.065 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.064+0000 7f768ccfb700 1 monclient: found mon.noname-a 2023-09-30T00:50:08.065 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.064+0000 7f768ccfb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:08.065 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.064+0000 7f768ccfb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:50:08.065 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.064+0000 7f768ccfb700 1 -- 172.21.15.40:0/990043595 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76880a74d0 con 0x7f76880b1830 2023-09-30T00:50:08.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.070+0000 7f768ccfb700 1 --2- 172.21.15.40:0/990043595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76880b1830 0x7f76880a5fb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f76780099e0 tx=0x7f7678009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d59b77904d3ac63 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f767ffff700 1 -- 172.21.15.40:0/990043595 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767800b7e0 con 0x7f76880b1830 2023-09-30T00:50:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f767ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f767ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:50:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f767ffff700 10 monclient: dump: 2023-09-30T00:50:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:08.077 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:08.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f767ffff700 10 monclient: _finish_auth 0 2023-09-30T00:50:08.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f767ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:08.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f768f760700 5 monclient: authenticate success, global_id 4530 2023-09-30T00:50:08.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f767ffff700 1 -- 172.21.15.40:0/990043595 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767800b940 con 0x7f76880b1830 2023-09-30T00:50:08.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f767ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:08.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f768f760700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:08.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f768f760700 4 set_mon_vals no callback set 2023-09-30T00:50:08.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f767ffff700 1 -- 172.21.15.40:0/990043595 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767800bce0 con 0x7f76880b1830 2023-09-30T00:50:08.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f767ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:08.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f767ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:50:08.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f767ffff700 10 monclient: dump: 2023-09-30T00:50:08.078 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:08.079 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:08.079 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:08.079 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:08.079 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:08.079 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:08.079 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:08.079 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:08.079 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:08.079 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f768f760700 10 monclient: shutdown 2023-09-30T00:50:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f768f760700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f768f760700 1 -- 172.21.15.40:0/990043595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76880b1830 msgr2=0x7f76880a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f768f760700 1 --2- 172.21.15.40:0/990043595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76880b1830 0x7f76880a5fb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f76780099e0 tx=0x7f7678009250 comp rx=0 tx=0).stop 2023-09-30T00:50:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f768f760700 1 -- 172.21.15.40:0/990043595 shutdown_connections 2023-09-30T00:50:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f768f760700 1 --2- 172.21.15.40:0/990043595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76880b1830 0x7f76880a5fb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.076+0000 7f768f760700 1 -- 172.21.15.40:0/990043595 >> 172.21.15.40:0/990043595 conn(0x7f76880631f0 msgr2=0x7f76880635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.077+0000 7f768f760700 1 -- 172.21.15.40:0/990043595 shutdown_connections 2023-09-30T00:50:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.077+0000 7f768f760700 1 -- 172.21.15.40:0/990043595 wait complete. 2023-09-30T00:50:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.077+0000 7f768f760700 10 monclient: build_initial_monmap 2023-09-30T00:50:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.077+0000 7f768f760700 10 monclient: monmap: 2023-09-30T00:50:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:08.083 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:08.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.077+0000 7f768f760700 1 librados: starting msgr at 2023-09-30T00:50:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.077+0000 7f768f760700 1 librados: starting objecter 2023-09-30T00:50:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.078+0000 7f768f760700 1 Processor -- start 2023-09-30T00:50:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.078+0000 7f768f760700 1 -- start start 2023-09-30T00:50:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.078+0000 7f768f760700 1 librados: setting wanted keys 2023-09-30T00:50:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.078+0000 7f768f760700 1 librados: calling monclient init 2023-09-30T00:50:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.078+0000 7f768f760700 10 monclient: init 2023-09-30T00:50:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.078+0000 7f768f760700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.078+0000 7f768f760700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:50:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.078+0000 7f768f760700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76880b1460 0x7f768813dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.078+0000 7f768f760700 10 monclient(hunting): picked mon.noname-c con 0x7f76880b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.078+0000 7f768f760700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76880b1830 0x7f76880a93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.078+0000 7f768f760700 10 monclient(hunting): picked mon.noname-a con 0x7f76880b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768f760700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f768813ac20 0x7f768814e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768f760700 10 monclient(hunting): picked mon.noname-b con 0x7f768813ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768f760700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768f760700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76880ab3f0 con 0x7f76880b1830 2023-09-30T00:50:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768f760700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768f760700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76880a9d30 con 0x7f768813ac20 2023-09-30T00:50:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768f760700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768f760700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76880aa1c0 con 0x7f76880b1460 2023-09-30T00:50:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768f760700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768f760700 10 monclient(hunting): authenticate will time out at 5469.380041s 2023-09-30T00:50:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768ccfb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76880b1830 0x7f76880a93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768d4fc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76880b1460 0x7f768813dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768dcfd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f768813ac20 0x7f768814e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768ccfb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76880b1830 0x7f76880a93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56688/0 (socket says 172.21.15.40:56688) 2023-09-30T00:50:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768ccfb700 1 -- 172.21.15.40:0/114830363 learned_addr learned my addr 172.21.15.40:0/114830363 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768ccfb700 10 monclient(hunting): get_auth_request con 0x7f76880b1830 auth_method 0 2023-09-30T00:50:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768ccfb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768ccfb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.079+0000 7f768ccfb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768d4fc700 10 monclient(hunting): get_auth_request con 0x7f76880b1460 auth_method 0 2023-09-30T00:50:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768d4fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768d4fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768d4fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768dcfd700 10 monclient(hunting): get_auth_request con 0x7f768813ac20 auth_method 0 2023-09-30T00:50:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768dcfd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768dcfd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:08.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768dcfd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:08.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768ccfb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:08.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768ccfb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:08.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768ccfb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:08.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768dcfd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:08.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768dcfd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:08.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768dcfd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:08.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768d4fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:08.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768d4fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:08.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.080+0000 7f768d4fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.084+0000 7f768d4fc700 10 monclient(hunting): handle_auth_done global_id 4475 payload 824 2023-09-30T00:50:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.084+0000 7f768d4fc700 1 -- 172.21.15.40:0/114830363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f768813ac20 msgr2=0x7f768814e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.084+0000 7f768d4fc700 1 --2- 172.21.15.40:0/114830363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f768813ac20 0x7f768814e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.084+0000 7f768d4fc700 1 -- 172.21.15.40:0/114830363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76880b1830 msgr2=0x7f76880a93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.084+0000 7f768d4fc700 1 --2- 172.21.15.40:0/114830363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76880b1830 0x7f76880a93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.084+0000 7f768d4fc700 10 monclient: _finish_hunting 0 2023-09-30T00:50:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.084+0000 7f768d4fc700 1 monclient: found mon.noname-c 2023-09-30T00:50:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.084+0000 7f768d4fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.084+0000 7f768d4fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:50:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.084+0000 7f768d4fc700 1 -- 172.21.15.40:0/114830363 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76780094f0 con 0x7f76880b1460 2023-09-30T00:50:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.085+0000 7f768dcfd700 1 --2- 172.21.15.40:0/114830363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f768813ac20 0x7f768814e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:50:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.087+0000 7f768d4fc700 1 --2- 172.21.15.40:0/114830363 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76880b1460 0x7f768813dce0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f768400bb30 tx=0x7f7684009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.087+0000 7f767d7fa700 10 client.?.objecter ms_handle_connect 0x7f76880b1460 2023-09-30T00:50:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.087+0000 7f767d7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:08.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f767d7fa700 1 -- 172.21.15.40:0/114830363 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768400c7c0 con 0x7f76880b1460 2023-09-30T00:50:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f767d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f767d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:50:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f767d7fa700 10 monclient: dump: 2023-09-30T00:50:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:08.093 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f767d7fa700 10 monclient: _finish_auth 0 2023-09-30T00:50:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f767d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f767d7fa700 1 -- 172.21.15.40:0/114830363 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768400c920 con 0x7f76880b1460 2023-09-30T00:50:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f768f760700 5 monclient: authenticate success, global_id 4475 2023-09-30T00:50:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f767d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f768f760700 10 monclient: _renew_subs 2023-09-30T00:50:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f767d7fa700 1 -- 172.21.15.40:0/114830363 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768400cce0 con 0x7f76880b1460 2023-09-30T00:50:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f768f760700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f768f760700 1 -- 172.21.15.40:0/114830363 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76880a9970 con 0x7f76880b1460 2023-09-30T00:50:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f767d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f767d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:50:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f767d7fa700 10 monclient: dump: 2023-09-30T00:50:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:08.095 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.091+0000 7f767f7fe700 4 set_mon_vals no callback set 2023-09-30T00:50:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.092+0000 7f768f760700 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.092+0000 7f768f760700 10 monclient: _renew_subs 2023-09-30T00:50:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.092+0000 7f768f760700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.092+0000 7f768f760700 1 -- 172.21.15.40:0/114830363 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768814e600 con 0x7f76880b1460 2023-09-30T00:50:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.092+0000 7f768f760700 1 librados: init done 2023-09-30T00:50:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.093+0000 7f768f760700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.093+0000 7f768f760700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.093+0000 7f768f760700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:08.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.093+0000 7f768f760700 1 -- 172.21.15.40:0/114830363 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7688064a00 con 0x7f76880b1460 2023-09-30T00:50:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.158+0000 7f767d7fa700 1 -- 172.21.15.40:0/114830363 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7684006d50 con 0x7f76880b1460 2023-09-30T00:50:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.158+0000 7f767d7fa700 1 --2- 172.21.15.40:0/114830363 >> v2:172.21.15.135:6800/116426 conn(0x7f766c068960 0x7f766c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.158+0000 7f767d7fa700 1 -- 172.21.15.40:0/114830363 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7684084260 con 0x7f76880b1460 2023-09-30T00:50:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.158+0000 7f767d7fa700 10 client.4475.objecter ms_dispatch 0x7f7688001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.158+0000 7f767d7fa700 3 client.4475.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.158+0000 7f767d7fa700 3 client.4475.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.158+0000 7f767d7fa700 20 client.4475.objecter dump_active .. 0 homeless 2023-09-30T00:50:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.158+0000 7f768ccfb700 1 --2- 172.21.15.40:0/114830363 >> v2:172.21.15.135:6800/116426 conn(0x7f766c068960 0x7f766c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.158+0000 7f768ccfb700 10 monclient: get_auth_request con 0x7f766c068960 auth_method 0 2023-09-30T00:50:08.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.159+0000 7f767d7fa700 10 client.4475.objecter ms_handle_connect 0x7f766c068960 2023-09-30T00:50:08.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.159+0000 7f768ccfb700 1 --2- 172.21.15.40:0/114830363 >> v2:172.21.15.135:6800/116426 conn(0x7f766c068960 0x7f766c06ae10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7688142720 tx=0x7f7678016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:08.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.395+0000 7f767d7fa700 1 -- 172.21.15.40:0/114830363 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7684051c80 con 0x7f76880b1460 2023-09-30T00:50:08.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.395+0000 7f767d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:08.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.395+0000 7f767d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:08.421 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:03:44.133 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:08.478 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:44.165 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:08.480 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:44.167 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:08.486 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:03:44.198 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:08.486 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:03:44.198 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:08.488 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:03:44.200 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:08.503 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:03:44.189 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:08.525 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:03:44.237 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:08.565 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:03:44.276 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:08.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.676+0000 7f768f760700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:08.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.676+0000 7f768f760700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:08.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.676+0000 7f768f760700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.676+0000 7f768f760700 1 -- 172.21.15.40:0/114830363 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f76880b3a90 con 0x7f76880b1460 2023-09-30T00:50:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.752+0000 7f767d7fa700 1 -- 172.21.15.40:0/114830363 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7684022080 con 0x7f76880b1460 2023-09-30T00:50:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.752+0000 7f767d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.752+0000 7f767d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:08.754 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:08.754 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:08.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 10 librados: watch_flush enter 2023-09-30T00:50:08.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 10 librados: watch_flush exit 2023-09-30T00:50:08.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 20 client.4475.objecter shutdown clearing up homeless session... 2023-09-30T00:50:08.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 10 client.4475.objecter successfully canceled tick 2023-09-30T00:50:08.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 1 -- 172.21.15.40:0/114830363 >> v2:172.21.15.135:6800/116426 conn(0x7f766c068960 msgr2=0x7f766c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:08.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 1 --2- 172.21.15.40:0/114830363 >> v2:172.21.15.135:6800/116426 conn(0x7f766c068960 0x7f766c06ae10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7688142720 tx=0x7f7678016000 comp rx=0 tx=0).stop 2023-09-30T00:50:08.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 10 monclient: shutdown 2023-09-30T00:50:08.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:08.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 1 -- 172.21.15.40:0/114830363 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76880b1460 msgr2=0x7f768813dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:08.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 1 --2- 172.21.15.40:0/114830363 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76880b1460 0x7f768813dce0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f768400bb30 tx=0x7f7684009780 comp rx=0 tx=0).stop 2023-09-30T00:50:08.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 1 -- 172.21.15.40:0/114830363 shutdown_connections 2023-09-30T00:50:08.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 1 --2- 172.21.15.40:0/114830363 >> v2:172.21.15.135:6800/116426 conn(0x7f766c068960 0x7f766c06ae10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:08.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 1 --2- 172.21.15.40:0/114830363 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f768813ac20 0x7f768814e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:08.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 1 --2- 172.21.15.40:0/114830363 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f76880b1460 0x7f768813dce0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:08.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 1 --2- 172.21.15.40:0/114830363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76880b1830 0x7f76880a93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:08.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 1 -- 172.21.15.40:0/114830363 >> 172.21.15.40:0/114830363 conn(0x7f76880631f0 msgr2=0x7f768813a030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:08.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 1 -- 172.21.15.40:0/114830363 shutdown_connections 2023-09-30T00:50:08.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 1 -- 172.21.15.40:0/114830363 wait complete. 2023-09-30T00:50:08.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:08.755+0000 7f768f760700 1 librados: shutdown 2023-09-30T00:50:11.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:11.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.912+0000 7f331e81c700 10 monclient: get_monmap_and_config 2023-09-30T00:50:11.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.912+0000 7f331e81c700 10 monclient: build_initial_monmap 2023-09-30T00:50:11.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.912+0000 7f331e81c700 10 monclient: monmap: 2023-09-30T00:50:11.913 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:11.913 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:11.913 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:11.913700+0000 2023-09-30T00:50:11.913 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:11.913700+0000 2023-09-30T00:50:11.914 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:11.914 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:11.914 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:11.914 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:11.914 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:11.914 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:11.914 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:11.914 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:11.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.913+0000 7f331e81c700 1 Processor -- start 2023-09-30T00:50:11.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.913+0000 7f331e81c700 1 -- start start 2023-09-30T00:50:11.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.913+0000 7f331e81c700 10 monclient: init 2023-09-30T00:50:11.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f331e81c700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:11.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f331e81c700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-09-30T00:50:11.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f331e81c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f33180b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:50:11.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f331e81c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33180b1830 0x7f33180a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:11.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f331e81c700 10 monclient(hunting): picked mon.noname-b con 0x7f33180b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:11.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f331e81c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f331813ac20 addr v1:172.21.15.135:6789/0 2023-09-30T00:50:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f331e81c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33180a5b20 con 0x7f33180b1460 2023-09-30T00:50:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f331e81c700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33180a71f0 con 0x7f331813ac20 2023-09-30T00:50:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f331e81c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f331e81c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3318141e40 con 0x7f33180b1830 2023-09-30T00:50:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f331e81c700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f331e81c700 10 monclient(hunting): authenticate will time out at 5473.215242s 2023-09-30T00:50:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f33177fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33180b1830 0x7f33180a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f331cdb9700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f331813ac20 0x7f331813b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:42810/0 (socket says 172.21.15.40:42810) 2023-09-30T00:50:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.914+0000 7f331cdb9700 1 -- 172.21.15.40:0/3615966105 learned_addr learned my addr 172.21.15.40:0/3615966105 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:50:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.915+0000 7f33177fe700 10 monclient(hunting): get_auth_request con 0x7f33180b1830 auth_method 0 2023-09-30T00:50:11.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.915+0000 7f33177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.915+0000 7f33177fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.915+0000 7f33177fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.916+0000 7f33177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.916+0000 7f33177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:11.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.916+0000 7f33177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.918+0000 7f3316ffd700 1 -- 172.21.15.40:0/3615966105 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496538716 0 0) 0x7f33180a71f0 con 0x7f331813ac20 2023-09-30T00:50:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.918+0000 7f3316ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.918+0000 7f3316ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.918+0000 7f3316ffd700 10 monclient(hunting): my global_id is 4591 2023-09-30T00:50:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.918+0000 7f3316ffd700 1 -- 172.21.15.40:0/3615966105 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3300003570 con 0x7f331813ac20 2023-09-30T00:50:11.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.920+0000 7f33177fe700 10 monclient(hunting): handle_auth_done global_id 4478 payload 274 2023-09-30T00:50:11.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.921+0000 7f33177fe700 1 -- 172.21.15.40:0/3615966105 >> v1:172.21.15.135:6789/0 conn(0x7f331813ac20 legacy=0x7f331813b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:11.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.921+0000 7f33177fe700 1 -- 172.21.15.40:0/3615966105 >> v1:172.21.15.40:6789/0 conn(0x7f33180b1460 legacy=0x7f3318141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:11.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.921+0000 7f33177fe700 10 monclient: _finish_hunting 0 2023-09-30T00:50:11.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.921+0000 7f33177fe700 1 monclient: found mon.noname-b 2023-09-30T00:50:11.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.921+0000 7f33177fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:11.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.921+0000 7f33177fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:50:11.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.921+0000 7f33177fe700 1 -- 172.21.15.40:0/3615966105 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33180a74d0 con 0x7f33180b1830 2023-09-30T00:50:11.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.925+0000 7f33177fe700 1 --2- 172.21.15.40:0/3615966105 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33180b1830 0x7f33180a5fb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f330c009910 tx=0x7f330c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64494f6380d28c60 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:11.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f3316ffd700 1 -- 172.21.15.40:0/3615966105 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c00b640 con 0x7f33180b1830 2023-09-30T00:50:11.931 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f3316ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:11.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f3316ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:11.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f3316ffd700 10 monclient: dump: 2023-09-30T00:50:11.932 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:11.932 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:11.932 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:11.932 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:11.932 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:11.932 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:11.932 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:11.932 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:11.932 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:11.932 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f3316ffd700 10 monclient: _finish_auth 0 2023-09-30T00:50:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f3316ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f3316ffd700 1 -- 172.21.15.40:0/3615966105 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330c00b7a0 con 0x7f33180b1830 2023-09-30T00:50:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f331e81c700 5 monclient: authenticate success, global_id 4478 2023-09-30T00:50:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f331e81c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:50:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f3316ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f331e81c700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f3316ffd700 1 -- 172.21.15.40:0/3615966105 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330c00bb60 con 0x7f33180b1830 2023-09-30T00:50:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f3316ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f331e81c700 4 set_mon_vals no callback set 2023-09-30T00:50:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f3316ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:50:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f3316ffd700 10 monclient: dump: 2023-09-30T00:50:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:11.934 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:11.935 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:11.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f331e81c700 10 monclient: shutdown 2023-09-30T00:50:11.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f331e81c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:11.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f331e81c700 1 -- 172.21.15.40:0/3615966105 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33180b1830 msgr2=0x7f33180a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:11.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f331e81c700 1 --2- 172.21.15.40:0/3615966105 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33180b1830 0x7f33180a5fb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f330c009910 tx=0x7f330c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:50:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f331e81c700 1 -- 172.21.15.40:0/3615966105 shutdown_connections 2023-09-30T00:50:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f331e81c700 1 --2- 172.21.15.40:0/3615966105 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33180b1830 0x7f33180a5fb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f331e81c700 1 -- 172.21.15.40:0/3615966105 >> 172.21.15.40:0/3615966105 conn(0x7f33180631f0 msgr2=0x7f33180635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f331e81c700 1 -- 172.21.15.40:0/3615966105 shutdown_connections 2023-09-30T00:50:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.931+0000 7f331e81c700 1 -- 172.21.15.40:0/3615966105 wait complete. 2023-09-30T00:50:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.932+0000 7f331e81c700 10 monclient: build_initial_monmap 2023-09-30T00:50:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.932+0000 7f331e81c700 10 monclient: monmap: 2023-09-30T00:50:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:11.939 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.932+0000 7f331e81c700 1 librados: starting msgr at 2023-09-30T00:50:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.932+0000 7f331e81c700 1 librados: starting objecter 2023-09-30T00:50:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.933+0000 7f331e81c700 1 Processor -- start 2023-09-30T00:50:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.933+0000 7f331e81c700 1 -- start start 2023-09-30T00:50:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.933+0000 7f331e81c700 1 librados: setting wanted keys 2023-09-30T00:50:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.933+0000 7f331e81c700 1 librados: calling monclient init 2023-09-30T00:50:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.933+0000 7f331e81c700 10 monclient: init 2023-09-30T00:50:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.933+0000 7f331e81c700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.933+0000 7f331e81c700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:50:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.933+0000 7f331e81c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33180b1460 0x7f331813e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.933+0000 7f331e81c700 10 monclient(hunting): picked mon.noname-c con 0x7f33180b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.933+0000 7f331e81c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33180b1830 0x7f33180a9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.933+0000 7f331e81c700 10 monclient(hunting): picked mon.noname-b con 0x7f33180b1830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.933+0000 7f331e81c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f331813ac20 0x7f33180a9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.933+0000 7f331e81c700 10 monclient(hunting): picked mon.noname-a con 0x7f331813ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.933+0000 7f331e81c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f331e81c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33180a95c0 con 0x7f331813ac20 2023-09-30T00:50:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f331e81c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f331e81c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331813ec50 con 0x7f33180b1830 2023-09-30T00:50:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f331e81c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f331e81c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331813ee10 con 0x7f33180b1460 2023-09-30T00:50:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f331e81c700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:11.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f331e81c700 10 monclient(hunting): authenticate will time out at 5473.235025s 2023-09-30T00:50:11.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f3317fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33180b1460 0x7f331813e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:11.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f331cdb9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f331813ac20 0x7f33180a9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:11.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f33177fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33180b1830 0x7f33180a9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:11.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f3317fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33180b1460 0x7f331813e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:39528/0 (socket says 172.21.15.40:39528) 2023-09-30T00:50:11.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f3317fff700 1 -- 172.21.15.40:0/3704751172 learned_addr learned my addr 172.21.15.40:0/3704751172 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:11.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f3317fff700 10 monclient(hunting): get_auth_request con 0x7f33180b1460 auth_method 0 2023-09-30T00:50:11.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f3317fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:11.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f3317fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:11.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.934+0000 7f3317fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:11.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f33177fe700 10 monclient(hunting): get_auth_request con 0x7f33180b1830 auth_method 0 2023-09-30T00:50:11.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f33177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:11.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f33177fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:11.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f33177fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:11.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f331cdb9700 10 monclient(hunting): get_auth_request con 0x7f331813ac20 auth_method 0 2023-09-30T00:50:11.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f331cdb9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:11.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f331cdb9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:11.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f331cdb9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:11.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f3317fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:11.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f3317fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:11.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f3317fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:11.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f33177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f33177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f33177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f331cdb9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f331cdb9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.935+0000 7f331cdb9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.939+0000 7f3317fff700 10 monclient(hunting): handle_auth_done global_id 4481 payload 824 2023-09-30T00:50:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.939+0000 7f3317fff700 1 -- 172.21.15.40:0/3704751172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33180b1830 msgr2=0x7f33180a9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.939+0000 7f3317fff700 1 --2- 172.21.15.40:0/3704751172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33180b1830 0x7f33180a9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.939+0000 7f3317fff700 1 -- 172.21.15.40:0/3704751172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f331813ac20 msgr2=0x7f33180a9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.939+0000 7f3317fff700 1 --2- 172.21.15.40:0/3704751172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f331813ac20 0x7f33180a9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.939+0000 7f3317fff700 10 monclient: _finish_hunting 0 2023-09-30T00:50:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.939+0000 7f3317fff700 1 monclient: found mon.noname-c 2023-09-30T00:50:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.939+0000 7f3317fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.939+0000 7f3317fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:50:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.939+0000 7f3317fff700 1 -- 172.21.15.40:0/3704751172 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c009420 con 0x7f33180b1460 2023-09-30T00:50:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.942+0000 7f3317fff700 1 --2- 172.21.15.40:0/3704751172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33180b1460 0x7f331813e440 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f330800b5e0 tx=0x7f33080090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.942+0000 7f32fffff700 10 client.?.objecter ms_handle_connect 0x7f33180b1460 2023-09-30T00:50:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.942+0000 7f32fffff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f32fffff700 1 -- 172.21.15.40:0/3704751172 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330800f510 con 0x7f33180b1460 2023-09-30T00:50:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f32fffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f32fffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:50:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f32fffff700 10 monclient: dump: 2023-09-30T00:50:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:11.949 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f32fffff700 10 monclient: _finish_auth 0 2023-09-30T00:50:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f32fffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f32fffff700 1 -- 172.21.15.40:0/3704751172 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330800f690 con 0x7f33180b1460 2023-09-30T00:50:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f331e81c700 5 monclient: authenticate success, global_id 4481 2023-09-30T00:50:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f331e81c700 10 monclient: _renew_subs 2023-09-30T00:50:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f331e81c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f331e81c700 1 -- 172.21.15.40:0/3704751172 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33180a79f0 con 0x7f33180b1460 2023-09-30T00:50:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f32fffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f32fffff700 1 -- 172.21.15.40:0/3704751172 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330800fa50 con 0x7f33180b1460 2023-09-30T00:50:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f32fffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f32fffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:50:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f32fffff700 10 monclient: dump: 2023-09-30T00:50:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:11.951 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f33167fc700 4 set_mon_vals no callback set 2023-09-30T00:50:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f331e81c700 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f331e81c700 10 monclient: _renew_subs 2023-09-30T00:50:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f331e81c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f331e81c700 1 -- 172.21.15.40:0/3704751172 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33180aa560 con 0x7f33180b1460 2023-09-30T00:50:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.947+0000 7f331e81c700 1 librados: init done 2023-09-30T00:50:11.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.948+0000 7f331e81c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:11.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.948+0000 7f331e81c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:11.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.948+0000 7f331e81c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:11.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.948+0000 7f331e81c700 1 -- 172.21.15.40:0/3704751172 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33180a9b00 con 0x7f33180b1460 2023-09-30T00:50:11.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.976+0000 7f32fffff700 1 -- 172.21.15.40:0/3704751172 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3308024430 con 0x7f33180b1460 2023-09-30T00:50:11.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.976+0000 7f32fffff700 1 --2- 172.21.15.40:0/3704751172 >> v2:172.21.15.135:6800/116426 conn(0x7f33040689e0 0x7f330406ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:11.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.976+0000 7f32fffff700 1 -- 172.21.15.40:0/3704751172 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3308084250 con 0x7f33180b1460 2023-09-30T00:50:11.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.976+0000 7f32fffff700 10 client.4481.objecter ms_dispatch 0x7f3318001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:11.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.976+0000 7f32fffff700 3 client.4481.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:11.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.976+0000 7f32fffff700 3 client.4481.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:11.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.976+0000 7f32fffff700 20 client.4481.objecter dump_active .. 0 homeless 2023-09-30T00:50:11.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.976+0000 7f33177fe700 1 --2- 172.21.15.40:0/3704751172 >> v2:172.21.15.135:6800/116426 conn(0x7f33040689e0 0x7f330406ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:11.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.976+0000 7f33177fe700 10 monclient: get_auth_request con 0x7f33040689e0 auth_method 0 2023-09-30T00:50:11.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.977+0000 7f33177fe700 1 --2- 172.21.15.40:0/3704751172 >> v2:172.21.15.135:6800/116426 conn(0x7f33040689e0 0x7f330406ae90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f330c0091e0 tx=0x7f330c016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:11.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:11.977+0000 7f32fffff700 10 client.4481.objecter ms_handle_connect 0x7f33040689e0 2023-09-30T00:50:12.269 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.268+0000 7f32fffff700 1 -- 172.21.15.40:0/3704751172 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3308051b70 con 0x7f33180b1460 2023-09-30T00:50:12.269 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.268+0000 7f32fffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:12.269 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.268+0000 7f32fffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:12.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.548+0000 7f331e81c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:12.549 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.548+0000 7f331e81c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:12.549 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.548+0000 7f331e81c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:12.549 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.548+0000 7f331e81c700 1 -- 172.21.15.40:0/3704751172 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f33180ab8a0 con 0x7f33180b1460 2023-09-30T00:50:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.554+0000 7f32fffff700 1 -- 172.21.15.40:0/3704751172 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3308055190 con 0x7f33180b1460 2023-09-30T00:50:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.554+0000 7f32fffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.554+0000 7f32fffff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:12.555 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:12.555 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:12.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.556+0000 7f331e81c700 10 librados: watch_flush enter 2023-09-30T00:50:12.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.556+0000 7f331e81c700 10 librados: watch_flush exit 2023-09-30T00:50:12.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.556+0000 7f331e81c700 20 client.4481.objecter shutdown clearing up homeless session... 2023-09-30T00:50:12.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.556+0000 7f331e81c700 10 client.4481.objecter successfully canceled tick 2023-09-30T00:50:12.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.556+0000 7f331e81c700 1 -- 172.21.15.40:0/3704751172 >> v2:172.21.15.135:6800/116426 conn(0x7f33040689e0 msgr2=0x7f330406ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:12.557 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.556+0000 7f331e81c700 1 --2- 172.21.15.40:0/3704751172 >> v2:172.21.15.135:6800/116426 conn(0x7f33040689e0 0x7f330406ae90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f330c0091e0 tx=0x7f330c016000 comp rx=0 tx=0).stop 2023-09-30T00:50:12.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.556+0000 7f331e81c700 10 monclient: shutdown 2023-09-30T00:50:12.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.556+0000 7f331e81c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:12.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.556+0000 7f331e81c700 1 -- 172.21.15.40:0/3704751172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33180b1460 msgr2=0x7f331813e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:12.558 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.556+0000 7f331e81c700 1 --2- 172.21.15.40:0/3704751172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33180b1460 0x7f331813e440 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f330800b5e0 tx=0x7f33080090f0 comp rx=0 tx=0).stop 2023-09-30T00:50:12.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.557+0000 7f331e81c700 1 -- 172.21.15.40:0/3704751172 shutdown_connections 2023-09-30T00:50:12.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.557+0000 7f331e81c700 1 --2- 172.21.15.40:0/3704751172 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f331813ac20 0x7f33180a9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:12.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.557+0000 7f331e81c700 1 --2- 172.21.15.40:0/3704751172 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f33180b1460 0x7f331813e440 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:12.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.557+0000 7f331e81c700 1 --2- 172.21.15.40:0/3704751172 >> v2:172.21.15.135:6800/116426 conn(0x7f33040689e0 0x7f330406ae90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:12.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.557+0000 7f331e81c700 1 --2- 172.21.15.40:0/3704751172 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f33180b1830 0x7f33180a9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:12.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.557+0000 7f331e81c700 1 -- 172.21.15.40:0/3704751172 >> 172.21.15.40:0/3704751172 conn(0x7f33180631f0 msgr2=0x7f33180b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:12.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.558+0000 7f331e81c700 1 -- 172.21.15.40:0/3704751172 shutdown_connections 2023-09-30T00:50:12.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.558+0000 7f331e81c700 1 -- 172.21.15.40:0/3704751172 wait complete. 2023-09-30T00:50:12.559 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:12.558+0000 7f331e81c700 1 librados: shutdown 2023-09-30T00:50:15.574 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.712+0000 7f0d66714700 10 monclient: get_monmap_and_config 2023-09-30T00:50:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.712+0000 7f0d66714700 10 monclient: build_initial_monmap 2023-09-30T00:50:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.712+0000 7f0d66714700 10 monclient: monmap: 2023-09-30T00:50:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:15.713240+0000 2023-09-30T00:50:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:15.713240+0000 2023-09-30T00:50:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:15.714 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:15.714 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.712+0000 7f0d66714700 1 Processor -- start 2023-09-30T00:50:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 1 -- start start 2023-09-30T00:50:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 10 monclient: init 2023-09-30T00:50:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-09-30T00:50:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d600b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:50:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0d600b1830 0x7f0d600a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 10 monclient(hunting): picked mon.noname-a con 0x7f0d600b1830 addr v2:172.21.15.40:3300/0 2023-09-30T00:50:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d6013ac20 0x7f0d6013b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 10 monclient(hunting): picked mon.noname-b con 0x7f0d6013ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d600a5b20 con 0x7f0d600b1460 2023-09-30T00:50:15.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:15.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d60141e40 con 0x7f0d600b1830 2023-09-30T00:50:15.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:15.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d600a64f0 con 0x7f0d6013ac20 2023-09-30T00:50:15.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:15.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.713+0000 7f0d66714700 10 monclient(hunting): authenticate will time out at 5477.014715s 2023-09-30T00:50:15.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.714+0000 7f0d5f7fe700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0d600b1830 0x7f0d600a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:15.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.714+0000 7f0d64cb1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d6013ac20 0x7f0d6013b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:15.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.714+0000 7f0d5ffff700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f0d600b1460 0x7f0d60141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:42814/0 (socket says 172.21.15.40:42814) 2023-09-30T00:50:15.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.714+0000 7f0d5ffff700 1 -- 172.21.15.40:0/648156065 learned_addr learned my addr 172.21.15.40:0/648156065 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:50:15.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.715+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request con 0x7f0d600b1830 auth_method 0 2023-09-30T00:50:15.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.715+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:15.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.715+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:15.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.715+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:15.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.715+0000 7f0d64cb1700 10 monclient(hunting): get_auth_request con 0x7f0d6013ac20 auth_method 0 2023-09-30T00:50:15.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.715+0000 7f0d64cb1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:15.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.715+0000 7f0d64cb1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:15.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.715+0000 7f0d64cb1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:15.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.715+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:15.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.715+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:15.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.715+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:15.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.716+0000 7f0d64cb1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:15.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.716+0000 7f0d64cb1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:15.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.716+0000 7f0d64cb1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:15.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.717+0000 7f0d5effd700 1 -- 172.21.15.40:0/648156065 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580705948 0 0) 0x7f0d600a5b20 con 0x7f0d600b1460 2023-09-30T00:50:15.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.717+0000 7f0d5effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:15.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.717+0000 7f0d5effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.717+0000 7f0d5effd700 10 monclient(hunting): my global_id is 4597 2023-09-30T00:50:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.717+0000 7f0d5effd700 1 -- 172.21.15.40:0/648156065 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d48003570 con 0x7f0d600b1460 2023-09-30T00:50:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.718+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_done global_id 4539 payload 274 2023-09-30T00:50:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.718+0000 7f0d5f7fe700 1 -- 172.21.15.40:0/648156065 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d6013ac20 msgr2=0x7f0d6013b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.718+0000 7f0d5f7fe700 1 --2- 172.21.15.40:0/648156065 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d6013ac20 0x7f0d6013b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.718+0000 7f0d5f7fe700 1 -- 172.21.15.40:0/648156065 >> v1:172.21.15.135:6789/0 conn(0x7f0d600b1460 legacy=0x7f0d60141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.718+0000 7f0d5f7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:50:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.718+0000 7f0d5f7fe700 1 monclient: found mon.noname-a 2023-09-30T00:50:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.718+0000 7f0d5f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:15.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.718+0000 7f0d5f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:50:15.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.718+0000 7f0d5f7fe700 1 -- 172.21.15.40:0/648156065 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d600a71f0 con 0x7f0d600b1830 2023-09-30T00:50:15.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.720+0000 7f0d5f7fe700 1 --2- 172.21.15.40:0/648156065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d600b1830 0x7f0d600a5fb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009910 tx=0x7f0d540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba4bf100b683ee2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:15.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d5effd700 1 -- 172.21.15.40:0/648156065 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d54011630 con 0x7f0d600b1830 2023-09-30T00:50:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d5effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:50:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d5effd700 10 monclient: dump: 2023-09-30T00:50:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:15.728 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d5effd700 10 monclient: _finish_auth 0 2023-09-30T00:50:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d5effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d66714700 5 monclient: authenticate success, global_id 4539 2023-09-30T00:50:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d66714700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:50:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d5effd700 1 -- 172.21.15.40:0/648156065 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d54011790 con 0x7f0d600b1830 2023-09-30T00:50:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d5effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d66714700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d66714700 4 set_mon_vals no callback set 2023-09-30T00:50:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d5effd700 1 -- 172.21.15.40:0/648156065 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d54011b30 con 0x7f0d600b1830 2023-09-30T00:50:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d5effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:50:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.725+0000 7f0d5effd700 10 monclient: dump: 2023-09-30T00:50:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:15.730 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:15.730 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:15.730 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:15.730 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:15.730 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:15.730 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:15.730 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:15.730 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:15.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.726+0000 7f0d66714700 10 monclient: shutdown 2023-09-30T00:50:15.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.726+0000 7f0d66714700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.726+0000 7f0d66714700 1 -- 172.21.15.40:0/648156065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d600b1830 msgr2=0x7f0d600a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.726+0000 7f0d66714700 1 --2- 172.21.15.40:0/648156065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d600b1830 0x7f0d600a5fb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009910 tx=0x7f0d540091a0 comp rx=0 tx=0).stop 2023-09-30T00:50:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.726+0000 7f0d66714700 1 -- 172.21.15.40:0/648156065 shutdown_connections 2023-09-30T00:50:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.726+0000 7f0d66714700 1 --2- 172.21.15.40:0/648156065 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d6013ac20 0x7f0d6013b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.726+0000 7f0d66714700 1 --2- 172.21.15.40:0/648156065 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d600b1830 0x7f0d600a5fb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.726+0000 7f0d66714700 1 -- 172.21.15.40:0/648156065 >> 172.21.15.40:0/648156065 conn(0x7f0d600631f0 msgr2=0x7f0d600635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.726+0000 7f0d66714700 1 -- 172.21.15.40:0/648156065 shutdown_connections 2023-09-30T00:50:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.726+0000 7f0d66714700 1 -- 172.21.15.40:0/648156065 wait complete. 2023-09-30T00:50:15.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.726+0000 7f0d66714700 10 monclient: build_initial_monmap 2023-09-30T00:50:15.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.726+0000 7f0d66714700 10 monclient: monmap: 2023-09-30T00:50:15.735 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:15.735 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:15.735 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:15.735 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:15.735 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:15.735 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:15.735 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:15.736 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:15.736 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:15.736 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:15.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.727+0000 7f0d66714700 1 librados: starting msgr at 2023-09-30T00:50:15.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.727+0000 7f0d66714700 1 librados: starting objecter 2023-09-30T00:50:15.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.727+0000 7f0d66714700 1 Processor -- start 2023-09-30T00:50:15.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.727+0000 7f0d66714700 1 -- start start 2023-09-30T00:50:15.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.727+0000 7f0d66714700 1 librados: setting wanted keys 2023-09-30T00:50:15.736 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.727+0000 7f0d66714700 1 librados: calling monclient init 2023-09-30T00:50:15.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.727+0000 7f0d66714700 10 monclient: init 2023-09-30T00:50:15.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:15.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:50:15.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d600b1460 0x7f0d601513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:15.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 10 monclient(hunting): picked mon.noname-b con 0x7f0d600b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:15.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d600b1830 0x7f0d60151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:15.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 10 monclient(hunting): picked mon.noname-c con 0x7f0d600b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:15.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d6013ac20 0x7f0d60138000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:15.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 10 monclient(hunting): picked mon.noname-a con 0x7f0d6013ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:15.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:15.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d60137b70 con 0x7f0d6013ac20 2023-09-30T00:50:15.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:15.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6013e680 con 0x7f0d600b1460 2023-09-30T00:50:15.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:15.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6013e840 con 0x7f0d600b1830 2023-09-30T00:50:15.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:15.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d66714700 10 monclient(hunting): authenticate will time out at 5477.029297s 2023-09-30T00:50:15.738 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d5ffff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d600b1460 0x7f0d601513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:15.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.728+0000 7f0d5f7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d600b1830 0x7f0d60151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:15.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d5ffff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d600b1460 0x7f0d601513f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:35460/0 (socket says 172.21.15.40:35460) 2023-09-30T00:50:15.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d5ffff700 1 -- 172.21.15.40:0/3278223992 learned_addr learned my addr 172.21.15.40:0/3278223992 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:15.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d5ffff700 10 monclient(hunting): get_auth_request con 0x7f0d600b1460 auth_method 0 2023-09-30T00:50:15.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:15.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d5ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:15.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d5ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:15.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request con 0x7f0d600b1830 auth_method 0 2023-09-30T00:50:15.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d5f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:15.739 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:15.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d5f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:15.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d64cb1700 1 --2- 172.21.15.40:0/3278223992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d6013ac20 0x7f0d60138000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:15.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d64cb1700 10 monclient(hunting): get_auth_request con 0x7f0d6013ac20 auth_method 0 2023-09-30T00:50:15.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d64cb1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:15.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d64cb1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:15.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d64cb1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:15.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d5ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:15.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d5ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:15.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.729+0000 7f0d5ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:15.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.730+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.730+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.730+0000 7f0d5f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.730+0000 7f0d64cb1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.730+0000 7f0d64cb1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.730+0000 7f0d64cb1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.733+0000 7f0d5ffff700 10 monclient(hunting): handle_auth_done global_id 4600 payload 824 2023-09-30T00:50:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.733+0000 7f0d5ffff700 1 -- 172.21.15.40:0/3278223992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d600b1830 msgr2=0x7f0d60151930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.733+0000 7f0d5ffff700 1 --2- 172.21.15.40:0/3278223992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d600b1830 0x7f0d60151930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:15.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.733+0000 7f0d5ffff700 1 -- 172.21.15.40:0/3278223992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d6013ac20 msgr2=0x7f0d60138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:15.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.733+0000 7f0d5ffff700 1 --2- 172.21.15.40:0/3278223992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d6013ac20 0x7f0d60138000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:15.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.733+0000 7f0d5ffff700 10 monclient: _finish_hunting 0 2023-09-30T00:50:15.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.733+0000 7f0d5ffff700 1 monclient: found mon.noname-b 2023-09-30T00:50:15.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.733+0000 7f0d5ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:15.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.733+0000 7f0d5ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:50:15.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.733+0000 7f0d5ffff700 1 -- 172.21.15.40:0/3278223992 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d54009420 con 0x7f0d600b1460 2023-09-30T00:50:15.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.735+0000 7f0d5ffff700 1 --2- 172.21.15.40:0/3278223992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d600b1460 0x7f0d601513f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000c590 tx=0x7f0d50007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:15.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.736+0000 7f0d47fff700 10 client.?.objecter ms_handle_connect 0x7f0d600b1460 2023-09-30T00:50:15.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.736+0000 7f0d47fff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:15.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d47fff700 1 -- 172.21.15.40:0/3278223992 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d50005b30 con 0x7f0d600b1460 2023-09-30T00:50:15.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d47fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:15.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d47fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:15.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d47fff700 10 monclient: dump: 2023-09-30T00:50:15.744 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:15.744 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:15.745 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:15.745 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:15.745 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:15.745 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:15.745 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:15.745 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:15.745 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:15.745 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:15.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d47fff700 10 monclient: _finish_auth 0 2023-09-30T00:50:15.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d47fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:15.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d47fff700 1 -- 172.21.15.40:0/3278223992 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d50005cb0 con 0x7f0d600b1460 2023-09-30T00:50:15.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d66714700 5 monclient: authenticate success, global_id 4600 2023-09-30T00:50:15.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d47fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:15.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d47fff700 1 -- 172.21.15.40:0/3278223992 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d50007320 con 0x7f0d600b1460 2023-09-30T00:50:15.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d66714700 10 monclient: _renew_subs 2023-09-30T00:50:15.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d66714700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:15.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d66714700 1 -- 172.21.15.40:0/3278223992 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d60151fb0 con 0x7f0d600b1460 2023-09-30T00:50:15.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d47fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:15.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d5e7fc700 4 set_mon_vals no callback set 2023-09-30T00:50:15.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d47fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:50:15.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d47fff700 10 monclient: dump: 2023-09-30T00:50:15.746 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:15.746 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:15.747 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:15.747 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:15.747 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:15.747 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:15.747 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:15.747 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:15.747 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:15.747 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:15.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d66714700 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:15.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d66714700 10 monclient: _renew_subs 2023-09-30T00:50:15.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d66714700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:15.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d66714700 1 -- 172.21.15.40:0/3278223992 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d60138780 con 0x7f0d600b1460 2023-09-30T00:50:15.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.741+0000 7f0d66714700 1 librados: init done 2023-09-30T00:50:15.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.742+0000 7f0d66714700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:15.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.742+0000 7f0d66714700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:15.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.742+0000 7f0d66714700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:15.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.742+0000 7f0d66714700 1 -- 172.21.15.40:0/3278223992 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d60064a00 con 0x7f0d600b1460 2023-09-30T00:50:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.994+0000 7f0d47fff700 1 -- 172.21.15.40:0/3278223992 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0d5001b3b0 con 0x7f0d600b1460 2023-09-30T00:50:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.995+0000 7f0d47fff700 1 --2- 172.21.15.40:0/3278223992 >> v2:172.21.15.135:6800/116426 conn(0x7f0d4c068990 0x7f0d4c06ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.995+0000 7f0d5f7fe700 1 --2- 172.21.15.40:0/3278223992 >> v2:172.21.15.135:6800/116426 conn(0x7f0d4c068990 0x7f0d4c06ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.995+0000 7f0d5f7fe700 10 monclient: get_auth_request con 0x7f0d4c068990 auth_method 0 2023-09-30T00:50:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.996+0000 7f0d47fff700 10 client.4600.objecter ms_handle_connect 0x7f0d4c068990 2023-09-30T00:50:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:15.996+0000 7f0d5f7fe700 1 --2- 172.21.15.40:0/3278223992 >> v2:172.21.15.135:6800/116426 conn(0x7f0d4c068990 0x7f0d4c06ae40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009860 tx=0x7f0d54017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:16.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.000+0000 7f0d47fff700 1 -- 172.21.15.40:0/3278223992 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0d50005e20 con 0x7f0d600b1460 2023-09-30T00:50:16.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.000+0000 7f0d47fff700 10 client.4600.objecter ms_dispatch 0x7f0d60001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:16.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.000+0000 7f0d47fff700 3 client.4600.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:16.000 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.000+0000 7f0d47fff700 3 client.4600.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:16.001 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.000+0000 7f0d47fff700 20 client.4600.objecter dump_active .. 0 homeless 2023-09-30T00:50:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.009+0000 7f0d47fff700 1 -- 172.21.15.40:0/3278223992 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0d50051050 con 0x7f0d600b1460 2023-09-30T00:50:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.009+0000 7f0d47fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.009+0000 7f0d47fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:16.271 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.271+0000 7f0d66714700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:16.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.271+0000 7f0d66714700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:16.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.271+0000 7f0d66714700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:16.272 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.271+0000 7f0d66714700 1 -- 172.21.15.40:0/3278223992 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0d600a6950 con 0x7f0d600b1460 2023-09-30T00:50:16.276 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.276+0000 7f0d47fff700 1 -- 172.21.15.40:0/3278223992 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0d50017100 con 0x7f0d600b1460 2023-09-30T00:50:16.276 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.276+0000 7f0d47fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:16.277 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.276+0000 7f0d47fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:16.277 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:16.277 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:16.279 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 10 librados: watch_flush enter 2023-09-30T00:50:16.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 10 librados: watch_flush exit 2023-09-30T00:50:16.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 20 client.4600.objecter shutdown clearing up homeless session... 2023-09-30T00:50:16.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 10 client.4600.objecter successfully canceled tick 2023-09-30T00:50:16.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 1 -- 172.21.15.40:0/3278223992 >> v2:172.21.15.135:6800/116426 conn(0x7f0d4c068990 msgr2=0x7f0d4c06ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:16.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 1 --2- 172.21.15.40:0/3278223992 >> v2:172.21.15.135:6800/116426 conn(0x7f0d4c068990 0x7f0d4c06ae40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009860 tx=0x7f0d54017000 comp rx=0 tx=0).stop 2023-09-30T00:50:16.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 10 monclient: shutdown 2023-09-30T00:50:16.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:16.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 1 -- 172.21.15.40:0/3278223992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d600b1460 msgr2=0x7f0d601513f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:16.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 1 --2- 172.21.15.40:0/3278223992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d600b1460 0x7f0d601513f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0d5000c590 tx=0x7f0d50007e00 comp rx=0 tx=0).stop 2023-09-30T00:50:16.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 1 -- 172.21.15.40:0/3278223992 shutdown_connections 2023-09-30T00:50:16.280 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 1 --2- 172.21.15.40:0/3278223992 >> v2:172.21.15.135:6800/116426 conn(0x7f0d4c068990 0x7f0d4c06ae40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:16.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 1 --2- 172.21.15.40:0/3278223992 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0d600b1460 0x7f0d601513f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:16.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 1 --2- 172.21.15.40:0/3278223992 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0d6013ac20 0x7f0d60138000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:16.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 1 --2- 172.21.15.40:0/3278223992 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0d600b1830 0x7f0d60151930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:16.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.278+0000 7f0d66714700 1 -- 172.21.15.40:0/3278223992 >> 172.21.15.40:0/3278223992 conn(0x7f0d600631f0 msgr2=0x7f0d60144900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:16.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.279+0000 7f0d66714700 1 -- 172.21.15.40:0/3278223992 shutdown_connections 2023-09-30T00:50:16.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.279+0000 7f0d66714700 1 -- 172.21.15.40:0/3278223992 wait complete. 2023-09-30T00:50:16.281 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:16.279+0000 7f0d66714700 1 librados: shutdown 2023-09-30T00:50:19.294 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:19.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.427+0000 7f3063019700 10 monclient: get_monmap_and_config 2023-09-30T00:50:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.427+0000 7f3063019700 10 monclient: build_initial_monmap 2023-09-30T00:50:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.427+0000 7f3063019700 10 monclient: monmap: 2023-09-30T00:50:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:19.428263+0000 2023-09-30T00:50:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:19.428263+0000 2023-09-30T00:50:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:19.428 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:19.429 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:19.429 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:19.429 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:19.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.428+0000 7f3063019700 1 Processor -- start 2023-09-30T00:50:19.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.428+0000 7f3063019700 1 -- start start 2023-09-30T00:50:19.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.428+0000 7f3063019700 10 monclient: init 2023-09-30T00:50:19.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.429+0000 7f3063019700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:19.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.429+0000 7f3063019700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-09-30T00:50:19.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.429+0000 7f3063019700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f305c0b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:50:19.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.429+0000 7f3063019700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f305c0b1830 addr v1:172.21.15.40:6789/0 2023-09-30T00:50:19.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.429+0000 7f3063019700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f305c0a5fb0 0x7f305c13b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:19.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.429+0000 7f3063019700 10 monclient(hunting): picked mon.noname-c con 0x7f305c0a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:50:19.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.429+0000 7f3063019700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f305c0a5b20 con 0x7f305c0b1830 2023-09-30T00:50:19.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.429+0000 7f3063019700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f305c0a71f0 con 0x7f305c0b1460 2023-09-30T00:50:19.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.429+0000 7f3063019700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:19.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.429+0000 7f3063019700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f305c141e40 con 0x7f305c0a5fb0 2023-09-30T00:50:19.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.429+0000 7f3063019700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:19.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.429+0000 7f3063019700 10 monclient(hunting): authenticate will time out at 5480.730267s 2023-09-30T00:50:19.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.429+0000 7f305bfff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f305c0b1830 0x7f305c13ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60144/0 (socket says 172.21.15.40:60144) 2023-09-30T00:50:19.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.429+0000 7f305bfff700 1 -- 172.21.15.40:0/998844773 learned_addr learned my addr 172.21.15.40:0/998844773 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:50:19.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.430+0000 7f30615b6700 1 --2- 172.21.15.40:0/998844773 >> v2:172.21.15.135:3300/0 conn(0x7f305c0a5fb0 0x7f305c13b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:19.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.430+0000 7f30615b6700 10 monclient(hunting): get_auth_request con 0x7f305c0a5fb0 auth_method 0 2023-09-30T00:50:19.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.430+0000 7f30615b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:19.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.430+0000 7f30615b6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:19.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.430+0000 7f30615b6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:19.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.431+0000 7f30615b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:19.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.431+0000 7f30615b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:19.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.431+0000 7f30615b6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:19.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.433+0000 7f305b7fe700 1 -- 172.21.15.40:0/998844773 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682993345 0 0) 0x7f305c0a5b20 con 0x7f305c0b1830 2023-09-30T00:50:19.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.433+0000 7f305b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.433+0000 7f305b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.433+0000 7f305b7fe700 10 monclient(hunting): my global_id is 4545 2023-09-30T00:50:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.433+0000 7f305b7fe700 1 -- 172.21.15.40:0/998844773 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3048003570 con 0x7f305c0b1830 2023-09-30T00:50:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.433+0000 7f30615b6700 10 monclient(hunting): handle_auth_done global_id 4603 payload 274 2023-09-30T00:50:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.433+0000 7f30615b6700 1 -- 172.21.15.40:0/998844773 >> v1:172.21.15.135:6789/0 conn(0x7f305c0b1460 legacy=0x7f305c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.433+0000 7f30615b6700 1 -- 172.21.15.40:0/998844773 >> v1:172.21.15.40:6789/0 conn(0x7f305c0b1830 legacy=0x7f305c13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.433+0000 7f30615b6700 10 monclient: _finish_hunting 0 2023-09-30T00:50:19.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.433+0000 7f30615b6700 1 monclient: found mon.noname-c 2023-09-30T00:50:19.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.434+0000 7f30615b6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:19.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.434+0000 7f30615b6700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:50:19.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.434+0000 7f30615b6700 1 -- 172.21.15.40:0/998844773 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c0a74d0 con 0x7f305c0a5fb0 2023-09-30T00:50:19.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.436+0000 7f30615b6700 1 --2- 172.21.15.40:0/998844773 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305c0a5fb0 0x7f305c13b330 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3054009e90 tx=0x7f3054009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=573b4de810039d23 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f305b7fe700 1 -- 172.21.15.40:0/998844773 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305400e070 con 0x7f305c0a5fb0 2023-09-30T00:50:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f305b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f305b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:50:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f305b7fe700 10 monclient: dump: 2023-09-30T00:50:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:19.443 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f305b7fe700 10 monclient: _finish_auth 0 2023-09-30T00:50:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f305b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f305b7fe700 1 -- 172.21.15.40:0/998844773 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305400bbb0 con 0x7f305c0a5fb0 2023-09-30T00:50:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f3063019700 5 monclient: authenticate success, global_id 4603 2023-09-30T00:50:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f3063019700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:50:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f305b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f3063019700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f305b7fe700 1 -- 172.21.15.40:0/998844773 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054012430 con 0x7f305c0a5fb0 2023-09-30T00:50:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f3063019700 4 set_mon_vals no callback set 2023-09-30T00:50:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f305b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f305b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:50:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.441+0000 7f305b7fe700 10 monclient: dump: 2023-09-30T00:50:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:19.445 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:19.445 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:19.445 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:19.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.442+0000 7f3063019700 10 monclient: shutdown 2023-09-30T00:50:19.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.442+0000 7f3063019700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:19.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.442+0000 7f3063019700 1 -- 172.21.15.40:0/998844773 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305c0a5fb0 msgr2=0x7f305c13b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:19.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.442+0000 7f3063019700 1 --2- 172.21.15.40:0/998844773 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305c0a5fb0 0x7f305c13b330 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3054009e90 tx=0x7f3054009780 comp rx=0 tx=0).stop 2023-09-30T00:50:19.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.442+0000 7f3063019700 1 -- 172.21.15.40:0/998844773 shutdown_connections 2023-09-30T00:50:19.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.442+0000 7f3063019700 1 --2- 172.21.15.40:0/998844773 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305c0a5fb0 0x7f305c13b330 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:19.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.442+0000 7f3063019700 1 -- 172.21.15.40:0/998844773 >> 172.21.15.40:0/998844773 conn(0x7f305c0631f0 msgr2=0x7f305c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:19.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.442+0000 7f3063019700 1 -- 172.21.15.40:0/998844773 shutdown_connections 2023-09-30T00:50:19.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.442+0000 7f3063019700 1 -- 172.21.15.40:0/998844773 wait complete. 2023-09-30T00:50:19.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.443+0000 7f3063019700 10 monclient: build_initial_monmap 2023-09-30T00:50:19.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.443+0000 7f3063019700 10 monclient: monmap: 2023-09-30T00:50:19.448 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:19.448 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:19.449 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.443+0000 7f3063019700 1 librados: starting msgr at 2023-09-30T00:50:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.443+0000 7f3063019700 1 librados: starting objecter 2023-09-30T00:50:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.443+0000 7f3063019700 1 Processor -- start 2023-09-30T00:50:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.443+0000 7f3063019700 1 -- start start 2023-09-30T00:50:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.443+0000 7f3063019700 1 librados: setting wanted keys 2023-09-30T00:50:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.443+0000 7f3063019700 1 librados: calling monclient init 2023-09-30T00:50:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.443+0000 7f3063019700 10 monclient: init 2023-09-30T00:50:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:50:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f305c0a5fb0 0x7f305c13e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 10 monclient(hunting): picked mon.noname-c con 0x7f305c0a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f305c0b1460 0x7f305c0a93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 10 monclient(hunting): picked mon.noname-a con 0x7f305c0b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305c0b1830 0x7f305c0aa050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 10 monclient(hunting): picked mon.noname-b con 0x7f305c0b1830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c0aba10 con 0x7f305c0b1460 2023-09-30T00:50:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c13ef80 con 0x7f305c0b1830 2023-09-30T00:50:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305c0a9a80 con 0x7f305c0a5fb0 2023-09-30T00:50:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f3063019700 10 monclient(hunting): authenticate will time out at 5480.745393s 2023-09-30T00:50:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.444+0000 7f305bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f305c0b1460 0x7f305c0a93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f30615b6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305c0b1830 0x7f305c0aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f3060db5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f305c0a5fb0 0x7f305c13e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f30615b6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305c0b1830 0x7f305c0aa050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:35476/0 (socket says 172.21.15.40:35476) 2023-09-30T00:50:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f30615b6700 1 -- 172.21.15.40:0/2081849256 learned_addr learned my addr 172.21.15.40:0/2081849256 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f30615b6700 10 monclient(hunting): get_auth_request con 0x7f305c0b1830 auth_method 0 2023-09-30T00:50:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f30615b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f30615b6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f30615b6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f305bfff700 10 monclient(hunting): get_auth_request con 0x7f305c0b1460 auth_method 0 2023-09-30T00:50:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f305bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f305bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f305bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f3060db5700 10 monclient(hunting): get_auth_request con 0x7f305c0a5fb0 auth_method 0 2023-09-30T00:50:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f3060db5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f3060db5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.445+0000 7f3060db5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.446+0000 7f30615b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.446+0000 7f30615b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.446+0000 7f30615b6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.446+0000 7f305bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.446+0000 7f305bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.446+0000 7f305bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.446+0000 7f3060db5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.446+0000 7f3060db5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.446+0000 7f3060db5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.449+0000 7f30615b6700 10 monclient(hunting): handle_auth_done global_id 4606 payload 824 2023-09-30T00:50:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.450+0000 7f30615b6700 1 -- 172.21.15.40:0/2081849256 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f305c0a5fb0 msgr2=0x7f305c13e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.450+0000 7f30615b6700 1 --2- 172.21.15.40:0/2081849256 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f305c0a5fb0 0x7f305c13e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.450+0000 7f30615b6700 1 -- 172.21.15.40:0/2081849256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f305c0b1460 msgr2=0x7f305c0a93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.450+0000 7f30615b6700 1 --2- 172.21.15.40:0/2081849256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f305c0b1460 0x7f305c0a93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.450+0000 7f30615b6700 10 monclient: _finish_hunting 0 2023-09-30T00:50:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.450+0000 7f30615b6700 1 monclient: found mon.noname-b 2023-09-30T00:50:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.450+0000 7f30615b6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.450+0000 7f30615b6700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:50:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.450+0000 7f30615b6700 1 -- 172.21.15.40:0/2081849256 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3054009a20 con 0x7f305c0b1830 2023-09-30T00:50:19.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.452+0000 7f30615b6700 1 --2- 172.21.15.40:0/2081849256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305c0b1830 0x7f305c0aa050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f30540099f0 tx=0x7f3054018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:19.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.452+0000 7f3058ff9700 10 client.?.objecter ms_handle_connect 0x7f305c0b1830 2023-09-30T00:50:19.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.452+0000 7f3058ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3058ff9700 1 -- 172.21.15.40:0/2081849256 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054027470 con 0x7f305c0b1830 2023-09-30T00:50:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3058ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3058ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3058ff9700 10 monclient: dump: 2023-09-30T00:50:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:19.458 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3058ff9700 10 monclient: _finish_auth 0 2023-09-30T00:50:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3058ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:19.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3058ff9700 1 -- 172.21.15.40:0/2081849256 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30540275d0 con 0x7f305c0b1830 2023-09-30T00:50:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3063019700 5 monclient: authenticate success, global_id 4606 2023-09-30T00:50:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3058ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3058ff9700 1 -- 172.21.15.40:0/2081849256 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054027990 con 0x7f305c0b1830 2023-09-30T00:50:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3063019700 10 monclient: _renew_subs 2023-09-30T00:50:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3063019700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3063019700 1 -- 172.21.15.40:0/2081849256 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305c0a79f0 con 0x7f305c0b1830 2023-09-30T00:50:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3058ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3058ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:50:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f3058ff9700 10 monclient: dump: 2023-09-30T00:50:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:19.459 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:19.460 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.456+0000 7f305affd700 4 set_mon_vals no callback set 2023-09-30T00:50:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.457+0000 7f3063019700 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.457+0000 7f3063019700 10 monclient: _renew_subs 2023-09-30T00:50:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.457+0000 7f3063019700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.457+0000 7f3063019700 1 -- 172.21.15.40:0/2081849256 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305c0aa750 con 0x7f305c0b1830 2023-09-30T00:50:19.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.457+0000 7f3063019700 1 librados: init done 2023-09-30T00:50:19.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.458+0000 7f3063019700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:19.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.458+0000 7f3063019700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:19.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.458+0000 7f3063019700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:19.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.458+0000 7f3063019700 1 -- 172.21.15.40:0/2081849256 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f305c0a9d70 con 0x7f305c0b1830 2023-09-30T00:50:19.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.708+0000 7f3058ff9700 1 -- 172.21.15.40:0/2081849256 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3054024070 con 0x7f305c0b1830 2023-09-30T00:50:19.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.709+0000 7f3058ff9700 1 --2- 172.21.15.40:0/2081849256 >> v2:172.21.15.135:6800/116426 conn(0x7f303c068890 0x7f303c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:19.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.709+0000 7f3060db5700 1 --2- 172.21.15.40:0/2081849256 >> v2:172.21.15.135:6800/116426 conn(0x7f303c068890 0x7f303c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:19.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.709+0000 7f3060db5700 10 monclient: get_auth_request con 0x7f303c068890 auth_method 0 2023-09-30T00:50:19.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.709+0000 7f3058ff9700 10 client.4606.objecter ms_handle_connect 0x7f303c068890 2023-09-30T00:50:19.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.709+0000 7f3060db5700 1 --2- 172.21.15.40:0/2081849256 >> v2:172.21.15.135:6800/116426 conn(0x7f303c068890 0x7f303c06ad40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f304c006fd0 tx=0x7f304c0090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:19.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.713+0000 7f3058ff9700 1 -- 172.21.15.40:0/2081849256 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f30540083a0 con 0x7f305c0b1830 2023-09-30T00:50:19.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.713+0000 7f3058ff9700 10 client.4606.objecter ms_dispatch 0x7f305c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:19.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.713+0000 7f3058ff9700 3 client.4606.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:19.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.713+0000 7f3058ff9700 3 client.4606.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:19.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.713+0000 7f3058ff9700 20 client.4606.objecter dump_active .. 0 homeless 2023-09-30T00:50:19.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.723+0000 7f3058ff9700 1 -- 172.21.15.40:0/2081849256 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f30540544b0 con 0x7f305c0b1830 2023-09-30T00:50:19.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.723+0000 7f3058ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:19.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.723+0000 7f3058ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:19.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.978+0000 7f3063019700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:19.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.978+0000 7f3063019700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:19.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.978+0000 7f3063019700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:19.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.978+0000 7f3063019700 1 -- 172.21.15.40:0/2081849256 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f305c0aa8e0 con 0x7f305c0b1830 2023-09-30T00:50:19.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.984+0000 7f3058ff9700 1 -- 172.21.15.40:0/2081849256 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f305402c100 con 0x7f305c0b1830 2023-09-30T00:50:19.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.984+0000 7f3058ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:19.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.984+0000 7f3058ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:19.984 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:19.984 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:19.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 10 librados: watch_flush enter 2023-09-30T00:50:19.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 10 librados: watch_flush exit 2023-09-30T00:50:19.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 20 client.4606.objecter shutdown clearing up homeless session... 2023-09-30T00:50:19.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 10 client.4606.objecter successfully canceled tick 2023-09-30T00:50:19.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 1 -- 172.21.15.40:0/2081849256 >> v2:172.21.15.135:6800/116426 conn(0x7f303c068890 msgr2=0x7f303c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:19.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 1 --2- 172.21.15.40:0/2081849256 >> v2:172.21.15.135:6800/116426 conn(0x7f303c068890 0x7f303c06ad40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f304c006fd0 tx=0x7f304c0090f0 comp rx=0 tx=0).stop 2023-09-30T00:50:19.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 10 monclient: shutdown 2023-09-30T00:50:19.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:19.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 1 -- 172.21.15.40:0/2081849256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305c0b1830 msgr2=0x7f305c0aa050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:19.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 1 --2- 172.21.15.40:0/2081849256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305c0b1830 0x7f305c0aa050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f30540099f0 tx=0x7f3054018000 comp rx=0 tx=0).stop 2023-09-30T00:50:19.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 1 -- 172.21.15.40:0/2081849256 shutdown_connections 2023-09-30T00:50:19.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 1 --2- 172.21.15.40:0/2081849256 >> v2:172.21.15.135:6800/116426 conn(0x7f303c068890 0x7f303c06ad40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:19.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 1 --2- 172.21.15.40:0/2081849256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f305c0b1830 0x7f305c0aa050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:19.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 1 --2- 172.21.15.40:0/2081849256 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f305c0a5fb0 0x7f305c13e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:19.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 1 --2- 172.21.15.40:0/2081849256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f305c0b1460 0x7f305c0a93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:19.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 1 -- 172.21.15.40:0/2081849256 >> 172.21.15.40:0/2081849256 conn(0x7f305c0631f0 msgr2=0x7f305c0b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:19.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 1 -- 172.21.15.40:0/2081849256 shutdown_connections 2023-09-30T00:50:19.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 1 -- 172.21.15.40:0/2081849256 wait complete. 2023-09-30T00:50:19.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:19.986+0000 7f3063019700 1 librados: shutdown 2023-09-30T00:50:23.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.140+0000 7f378b511700 10 monclient: get_monmap_and_config 2023-09-30T00:50:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.140+0000 7f378b511700 10 monclient: build_initial_monmap 2023-09-30T00:50:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.140+0000 7f378b511700 10 monclient: monmap: 2023-09-30T00:50:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:23.141484+0000 2023-09-30T00:50:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:23.141484+0000 2023-09-30T00:50:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:23.142 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:23.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.141+0000 7f378b511700 1 Processor -- start 2023-09-30T00:50:23.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.141+0000 7f378b511700 1 -- start start 2023-09-30T00:50:23.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.141+0000 7f378b511700 10 monclient: init 2023-09-30T00:50:23.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.142+0000 7f378b511700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.142+0000 7f378b511700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-09-30T00:50:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.142+0000 7f378b511700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37840b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:50:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.142+0000 7f378b511700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f37840b1830 0x7f37840a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.142+0000 7f378b511700 10 monclient(hunting): picked mon.noname-a con 0x7f37840b1830 addr v2:172.21.15.40:3300/0 2023-09-30T00:50:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.142+0000 7f378b511700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f378413ac20 0x7f378413b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.142+0000 7f378b511700 10 monclient(hunting): picked mon.noname-b con 0x7f378413ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.142+0000 7f378b511700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37840a5b20 con 0x7f37840b1460 2023-09-30T00:50:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.142+0000 7f378b511700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.142+0000 7f378b511700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3784141e40 con 0x7f37840b1830 2023-09-30T00:50:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.142+0000 7f378b511700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.142+0000 7f378b511700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37840a64f0 con 0x7f378413ac20 2023-09-30T00:50:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.142+0000 7f378b511700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.142+0000 7f378b511700 10 monclient(hunting): authenticate will time out at 5484.443509s 2023-09-30T00:50:23.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.143+0000 7f3789aae700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f378413ac20 0x7f378413b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:23.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.143+0000 7f37892ad700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f37840b1460 0x7f3784141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:45772/0 (socket says 172.21.15.40:45772) 2023-09-30T00:50:23.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.143+0000 7f37892ad700 1 -- 172.21.15.40:0/3308064004 learned_addr learned my addr 172.21.15.40:0/3308064004 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:50:23.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.143+0000 7f3789aae700 10 monclient(hunting): get_auth_request con 0x7f378413ac20 auth_method 0 2023-09-30T00:50:23.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.143+0000 7f3789aae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:23.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.143+0000 7f3789aae700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:23.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.143+0000 7f3789aae700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:23.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.143+0000 7f3788aac700 1 --2- 172.21.15.40:0/3308064004 >> v2:172.21.15.40:3300/0 conn(0x7f37840b1830 0x7f37840a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:23.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.144+0000 7f3789aae700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:23.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.144+0000 7f3789aae700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:23.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.144+0000 7f3789aae700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:23.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.145+0000 7f3788aac700 10 monclient(hunting): get_auth_request con 0x7f37840b1830 auth_method 0 2023-09-30T00:50:23.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.145+0000 7f3788aac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:23.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.145+0000 7f3788aac700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:23.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.145+0000 7f3788aac700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:23.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.146+0000 7f377bfff700 1 -- 172.21.15.40:0/3308064004 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924863113 0 0) 0x7f37840a5b20 con 0x7f37840b1460 2023-09-30T00:50:23.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.146+0000 7f377bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.146+0000 7f377bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.146+0000 7f377bfff700 10 monclient(hunting): my global_id is 4609 2023-09-30T00:50:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.146+0000 7f377bfff700 1 -- 172.21.15.40:0/3308064004 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3770003570 con 0x7f37840b1460 2023-09-30T00:50:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.146+0000 7f3789aae700 10 monclient(hunting): handle_auth_done global_id 4493 payload 274 2023-09-30T00:50:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.146+0000 7f3789aae700 1 -- 172.21.15.40:0/3308064004 >> v2:172.21.15.40:3300/0 conn(0x7f37840b1830 msgr2=0x7f37840a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.146+0000 7f3789aae700 1 --2- 172.21.15.40:0/3308064004 >> v2:172.21.15.40:3300/0 conn(0x7f37840b1830 0x7f37840a5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.146+0000 7f3789aae700 1 -- 172.21.15.40:0/3308064004 >> v1:172.21.15.135:6789/0 conn(0x7f37840b1460 legacy=0x7f3784141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.146+0000 7f3789aae700 10 monclient: _finish_hunting 0 2023-09-30T00:50:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.146+0000 7f3789aae700 1 monclient: found mon.noname-b 2023-09-30T00:50:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.146+0000 7f3789aae700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.146+0000 7f3789aae700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:50:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.146+0000 7f3789aae700 1 -- 172.21.15.40:0/3308064004 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37840a71f0 con 0x7f378413ac20 2023-09-30T00:50:23.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.149+0000 7f3789aae700 1 --2- 172.21.15.40:0/3308064004 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f378413ac20 0x7f378413b080 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f377c009e60 tx=0x7f377c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40858405f92842de server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f377bfff700 1 -- 172.21.15.40:0/3308064004 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c00e070 con 0x7f378413ac20 2023-09-30T00:50:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f377bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f377bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f377bfff700 10 monclient: dump: 2023-09-30T00:50:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:23.155 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f377bfff700 10 monclient: _finish_auth 0 2023-09-30T00:50:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f377bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f377bfff700 1 -- 172.21.15.40:0/3308064004 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377c00bb60 con 0x7f378413ac20 2023-09-30T00:50:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f377bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f377bfff700 1 -- 172.21.15.40:0/3308064004 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c012430 con 0x7f378413ac20 2023-09-30T00:50:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f378b511700 5 monclient: authenticate success, global_id 4493 2023-09-30T00:50:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f378b511700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f377bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f378b511700 4 set_mon_vals no callback set 2023-09-30T00:50:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f377bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:50:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.153+0000 7f377bfff700 10 monclient: dump: 2023-09-30T00:50:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:23.157 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.154+0000 7f378b511700 10 monclient: shutdown 2023-09-30T00:50:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.154+0000 7f378b511700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.154+0000 7f378b511700 1 -- 172.21.15.40:0/3308064004 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f378413ac20 msgr2=0x7f378413b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.154+0000 7f378b511700 1 --2- 172.21.15.40:0/3308064004 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f378413ac20 0x7f378413b080 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f377c009e60 tx=0x7f377c009780 comp rx=0 tx=0).stop 2023-09-30T00:50:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.154+0000 7f378b511700 1 -- 172.21.15.40:0/3308064004 shutdown_connections 2023-09-30T00:50:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.154+0000 7f378b511700 1 --2- 172.21.15.40:0/3308064004 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f378413ac20 0x7f378413b080 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.154+0000 7f378b511700 1 --2- 172.21.15.40:0/3308064004 >> v2:172.21.15.40:3300/0 conn(0x7f37840b1830 0x7f37840a5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.154+0000 7f378b511700 1 -- 172.21.15.40:0/3308064004 >> 172.21.15.40:0/3308064004 conn(0x7f37840631f0 msgr2=0x7f37840635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.154+0000 7f378b511700 1 -- 172.21.15.40:0/3308064004 shutdown_connections 2023-09-30T00:50:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.154+0000 7f378b511700 1 -- 172.21.15.40:0/3308064004 wait complete. 2023-09-30T00:50:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.155+0000 7f378b511700 10 monclient: build_initial_monmap 2023-09-30T00:50:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.155+0000 7f378b511700 10 monclient: monmap: 2023-09-30T00:50:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:23.161 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.155+0000 7f378b511700 1 librados: starting msgr at 2023-09-30T00:50:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.155+0000 7f378b511700 1 librados: starting objecter 2023-09-30T00:50:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.155+0000 7f378b511700 1 Processor -- start 2023-09-30T00:50:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 1 -- start start 2023-09-30T00:50:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 1 librados: setting wanted keys 2023-09-30T00:50:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 1 librados: calling monclient init 2023-09-30T00:50:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 10 monclient: init 2023-09-30T00:50:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:50:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37840b1460 0x7f3784150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 10 monclient(hunting): picked mon.noname-b con 0x7f37840b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37840b1830 0x7f3784139af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 10 monclient(hunting): picked mon.noname-c con 0x7f37840b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f378413ac20 0x7f378413a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:23.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 10 monclient(hunting): picked mon.noname-a con 0x7f378413ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37840ae550 con 0x7f378413ac20 2023-09-30T00:50:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37840ac120 con 0x7f37840b1460 2023-09-30T00:50:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37840ac400 con 0x7f37840b1830 2023-09-30T00:50:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.156+0000 7f378b511700 10 monclient(hunting): authenticate will time out at 5484.457925s 2023-09-30T00:50:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.157+0000 7f3788aac700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37840b1830 0x7f3784139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.157+0000 7f37892ad700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37840b1460 0x7f3784150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.157+0000 7f3789aae700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f378413ac20 0x7f378413a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f37892ad700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37840b1460 0x7f3784150aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:47610/0 (socket says 172.21.15.40:47610) 2023-09-30T00:50:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f37892ad700 1 -- 172.21.15.40:0/2684637722 learned_addr learned my addr 172.21.15.40:0/2684637722 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f37892ad700 10 monclient(hunting): get_auth_request con 0x7f37840b1460 auth_method 0 2023-09-30T00:50:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f37892ad700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f37892ad700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f37892ad700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f3789aae700 10 monclient(hunting): get_auth_request con 0x7f378413ac20 auth_method 0 2023-09-30T00:50:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f3789aae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f3789aae700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f3789aae700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f3788aac700 10 monclient(hunting): get_auth_request con 0x7f37840b1830 auth_method 0 2023-09-30T00:50:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f3788aac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f3788aac700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f3788aac700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f3789aae700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f3789aae700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f3789aae700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f3788aac700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f3788aac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f3788aac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f37892ad700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f37892ad700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.158+0000 7f37892ad700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.162+0000 7f37892ad700 10 monclient(hunting): handle_auth_done global_id 4612 payload 824 2023-09-30T00:50:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.162+0000 7f37892ad700 1 -- 172.21.15.40:0/2684637722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37840b1830 msgr2=0x7f3784139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.162+0000 7f37892ad700 1 --2- 172.21.15.40:0/2684637722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37840b1830 0x7f3784139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.162+0000 7f37892ad700 1 -- 172.21.15.40:0/2684637722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f378413ac20 msgr2=0x7f378413a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.162+0000 7f37892ad700 1 --2- 172.21.15.40:0/2684637722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f378413ac20 0x7f378413a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.162+0000 7f37892ad700 10 monclient: _finish_hunting 0 2023-09-30T00:50:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.162+0000 7f37892ad700 1 monclient: found mon.noname-b 2023-09-30T00:50:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.162+0000 7f37892ad700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.162+0000 7f37892ad700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:50:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.162+0000 7f37892ad700 1 -- 172.21.15.40:0/2684637722 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c009a20 con 0x7f37840b1460 2023-09-30T00:50:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.165+0000 7f37892ad700 1 --2- 172.21.15.40:0/2684637722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37840b1460 0x7f3784150aa0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f378000c7f0 tx=0x7f3780007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.165+0000 7f37797fa700 10 client.?.objecter ms_handle_connect 0x7f37840b1460 2023-09-30T00:50:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.165+0000 7f37797fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f37797fa700 1 -- 172.21.15.40:0/2684637722 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780005cd0 con 0x7f37840b1460 2023-09-30T00:50:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f37797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f37797fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f37797fa700 10 monclient: dump: 2023-09-30T00:50:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:23.171 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:23.172 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f37797fa700 10 monclient: _finish_auth 0 2023-09-30T00:50:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f37797fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f37797fa700 1 -- 172.21.15.40:0/2684637722 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3780005e50 con 0x7f37840b1460 2023-09-30T00:50:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f37797fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f37797fa700 1 -- 172.21.15.40:0/2684637722 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780007320 con 0x7f37840b1460 2023-09-30T00:50:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f378b511700 5 monclient: authenticate success, global_id 4612 2023-09-30T00:50:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f378b511700 10 monclient: _renew_subs 2023-09-30T00:50:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f378b511700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:23.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f378b511700 1 -- 172.21.15.40:0/2684637722 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37840ab4e0 con 0x7f37840b1460 2023-09-30T00:50:23.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f37797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:23.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f37797fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:50:23.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f37797fa700 10 monclient: dump: 2023-09-30T00:50:23.173 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:23.173 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:23.173 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:23.173 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:23.173 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:23.173 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:23.173 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:23.173 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:23.173 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:23.173 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:23.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.169+0000 7f377b7fe700 4 set_mon_vals no callback set 2023-09-30T00:50:23.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.170+0000 7f378b511700 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:23.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.170+0000 7f378b511700 10 monclient: _renew_subs 2023-09-30T00:50:23.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.170+0000 7f378b511700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:23.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.170+0000 7f378b511700 1 -- 172.21.15.40:0/2684637722 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37841514f0 con 0x7f37840b1460 2023-09-30T00:50:23.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.170+0000 7f378b511700 1 librados: init done 2023-09-30T00:50:23.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.170+0000 7f378b511700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:23.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.170+0000 7f378b511700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:23.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.170+0000 7f378b511700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:23.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.170+0000 7f378b511700 1 -- 172.21.15.40:0/2684637722 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3784151710 con 0x7f37840b1460 2023-09-30T00:50:23.191 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.190+0000 7f37797fa700 1 -- 172.21.15.40:0/2684637722 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f378001b3b0 con 0x7f37840b1460 2023-09-30T00:50:23.191 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.191+0000 7f37797fa700 1 --2- 172.21.15.40:0/2684637722 >> v2:172.21.15.135:6800/116426 conn(0x7f3768068960 0x7f376806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:23.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.191+0000 7f3788aac700 1 --2- 172.21.15.40:0/2684637722 >> v2:172.21.15.135:6800/116426 conn(0x7f3768068960 0x7f376806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:23.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.191+0000 7f3788aac700 10 monclient: get_auth_request con 0x7f3768068960 auth_method 0 2023-09-30T00:50:23.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.192+0000 7f3788aac700 1 --2- 172.21.15.40:0/2684637722 >> v2:172.21.15.135:6800/116426 conn(0x7f3768068960 0x7f376806ae10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f37840ade10 tx=0x7f3774008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:23.192 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.192+0000 7f37797fa700 10 client.4612.objecter ms_handle_connect 0x7f3768068960 2023-09-30T00:50:23.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.425+0000 7f37797fa700 1 -- 172.21.15.40:0/2684637722 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3780055660 con 0x7f37840b1460 2023-09-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.425+0000 7f37797fa700 10 client.4612.objecter ms_dispatch 0x7f3784001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.425+0000 7f37797fa700 3 client.4612.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.425+0000 7f37797fa700 3 client.4612.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:23.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.425+0000 7f37797fa700 20 client.4612.objecter dump_active .. 0 homeless 2023-09-30T00:50:23.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.436+0000 7f37797fa700 1 -- 172.21.15.40:0/2684637722 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f37800512a0 con 0x7f37840b1460 2023-09-30T00:50:23.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.436+0000 7f37797fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:23.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.436+0000 7f37797fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:23.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.695+0000 7f378b511700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:23.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.695+0000 7f378b511700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:23.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.695+0000 7f378b511700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:23.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.695+0000 7f378b511700 1 -- 172.21.15.40:0/2684637722 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3784064a00 con 0x7f37840b1460 2023-09-30T00:50:23.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.699+0000 7f37797fa700 1 -- 172.21.15.40:0/2684637722 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3780017100 con 0x7f37840b1460 2023-09-30T00:50:23.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.699+0000 7f37797fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:23.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.699+0000 7f37797fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:23.700 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:23.700 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:23.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 10 librados: watch_flush enter 2023-09-30T00:50:23.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 10 librados: watch_flush exit 2023-09-30T00:50:23.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 20 client.4612.objecter shutdown clearing up homeless session... 2023-09-30T00:50:23.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 10 client.4612.objecter successfully canceled tick 2023-09-30T00:50:23.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 1 -- 172.21.15.40:0/2684637722 >> v2:172.21.15.135:6800/116426 conn(0x7f3768068960 msgr2=0x7f376806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:23.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 1 --2- 172.21.15.40:0/2684637722 >> v2:172.21.15.135:6800/116426 conn(0x7f3768068960 0x7f376806ae10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f37840ade10 tx=0x7f3774008040 comp rx=0 tx=0).stop 2023-09-30T00:50:23.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 10 monclient: shutdown 2023-09-30T00:50:23.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:23.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 1 -- 172.21.15.40:0/2684637722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37840b1460 msgr2=0x7f3784150aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:23.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 1 --2- 172.21.15.40:0/2684637722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37840b1460 0x7f3784150aa0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f378000c7f0 tx=0x7f3780007e00 comp rx=0 tx=0).stop 2023-09-30T00:50:23.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 1 -- 172.21.15.40:0/2684637722 shutdown_connections 2023-09-30T00:50:23.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 1 --2- 172.21.15.40:0/2684637722 >> v2:172.21.15.135:6800/116426 conn(0x7f3768068960 0x7f376806ae10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:23.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 1 --2- 172.21.15.40:0/2684637722 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37840b1460 0x7f3784150aa0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:23.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 1 --2- 172.21.15.40:0/2684637722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f378413ac20 0x7f378413a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:23.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 1 --2- 172.21.15.40:0/2684637722 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37840b1830 0x7f3784139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:23.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 1 -- 172.21.15.40:0/2684637722 >> 172.21.15.40:0/2684637722 conn(0x7f37840631f0 msgr2=0x7f37840afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:23.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 1 -- 172.21.15.40:0/2684637722 shutdown_connections 2023-09-30T00:50:23.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 1 -- 172.21.15.40:0/2684637722 wait complete. 2023-09-30T00:50:23.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:23.702+0000 7f378b511700 1 librados: shutdown 2023-09-30T00:50:26.717 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:26.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.856+0000 7f4a9c876700 10 monclient: get_monmap_and_config 2023-09-30T00:50:26.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.856+0000 7f4a9c876700 10 monclient: build_initial_monmap 2023-09-30T00:50:26.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.856+0000 7f4a9c876700 10 monclient: monmap: 2023-09-30T00:50:26.858 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:26.858 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:26.858 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:26.857878+0000 2023-09-30T00:50:26.858 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:26.857878+0000 2023-09-30T00:50:26.858 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:26.858 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:26.858 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:26.858 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:26.858 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:26.859 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:26.859 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:26.859 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:26.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.857+0000 7f4a9c876700 1 Processor -- start 2023-09-30T00:50:26.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.857+0000 7f4a9c876700 1 -- start start 2023-09-30T00:50:26.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.857+0000 7f4a9c876700 10 monclient: init 2023-09-30T00:50:26.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.858+0000 7f4a9c876700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:26.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.858+0000 7f4a9c876700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-09-30T00:50:26.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.858+0000 7f4a9c876700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a980b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:50:26.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.858+0000 7f4a9c876700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a980b1830 addr v1:172.21.15.40:6789/0 2023-09-30T00:50:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.858+0000 7f4a9c876700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a980a5fb0 0x7f4a9813b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.858+0000 7f4a9c876700 10 monclient(hunting): picked mon.noname-a con 0x7f4a980a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:50:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.858+0000 7f4a9c876700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a980a5b20 con 0x7f4a980b1830 2023-09-30T00:50:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.858+0000 7f4a9c876700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a980a71f0 con 0x7f4a980b1460 2023-09-30T00:50:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.858+0000 7f4a9c876700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.858+0000 7f4a9c876700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a98141e40 con 0x7f4a980a5fb0 2023-09-30T00:50:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.858+0000 7f4a9c876700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.858+0000 7f4a9c876700 10 monclient(hunting): authenticate will time out at 5488.159413s 2023-09-30T00:50:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.859+0000 7f4a96d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a980a5fb0 0x7f4a9813b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.859+0000 7f4a9659c700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f4a980b1460 0x7f4a98141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:45782/0 (socket says 172.21.15.40:45782) 2023-09-30T00:50:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.859+0000 7f4a9659c700 1 -- 172.21.15.40:0/1697368288 learned_addr learned my addr 172.21.15.40:0/1697368288 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:50:26.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.859+0000 7f4a96d9d700 10 monclient(hunting): get_auth_request con 0x7f4a980a5fb0 auth_method 0 2023-09-30T00:50:26.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.859+0000 7f4a96d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:26.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.859+0000 7f4a96d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:26.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.859+0000 7f4a96d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:26.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.860+0000 7f4a96d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:26.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.860+0000 7f4a96d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:26.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.860+0000 7f4a96d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:26.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.862+0000 7f4a96d9d700 10 monclient(hunting): handle_auth_done global_id 4557 payload 274 2023-09-30T00:50:26.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.862+0000 7f4a96d9d700 1 -- 172.21.15.40:0/1697368288 >> v1:172.21.15.135:6789/0 conn(0x7f4a980b1460 legacy=0x7f4a98141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:26.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.862+0000 7f4a96d9d700 1 -- 172.21.15.40:0/1697368288 >> v1:172.21.15.40:6789/0 conn(0x7f4a980b1830 legacy=0x7f4a9813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:26.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.862+0000 7f4a96d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:50:26.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.862+0000 7f4a96d9d700 1 monclient: found mon.noname-a 2023-09-30T00:50:26.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.862+0000 7f4a96d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:26.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.862+0000 7f4a96d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:50:26.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.862+0000 7f4a96d9d700 1 -- 172.21.15.40:0/1697368288 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a980a74d0 con 0x7f4a980a5fb0 2023-09-30T00:50:26.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.865+0000 7f4a96d9d700 1 --2- 172.21.15.40:0/1697368288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a980a5fb0 0x7f4a9813b330 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4a90009e90 tx=0x7f4a90009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc31ec267249cb74 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.870+0000 7f4a9559a700 1 -- 172.21.15.40:0/1697368288 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9000e070 con 0x7f4a980a5fb0 2023-09-30T00:50:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.870+0000 7f4a9559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.870+0000 7f4a9559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:50:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.870+0000 7f4a9559a700 10 monclient: dump: 2023-09-30T00:50:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:26.872 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9559a700 10 monclient: _finish_auth 0 2023-09-30T00:50:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9559a700 1 -- 172.21.15.40:0/1697368288 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9000bbb0 con 0x7f4a980a5fb0 2023-09-30T00:50:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9c876700 5 monclient: authenticate success, global_id 4557 2023-09-30T00:50:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9c876700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:50:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9c876700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9c876700 4 set_mon_vals no callback set 2023-09-30T00:50:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9559a700 1 -- 172.21.15.40:0/1697368288 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90012430 con 0x7f4a980a5fb0 2023-09-30T00:50:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:50:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9559a700 10 monclient: dump: 2023-09-30T00:50:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:26.873 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:26.874 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:26.874 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:26.874 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:26.874 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:26.874 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:26.874 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:26.874 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:26.874 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:26.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9c876700 10 monclient: shutdown 2023-09-30T00:50:26.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9c876700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:26.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9c876700 1 -- 172.21.15.40:0/1697368288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a980a5fb0 msgr2=0x7f4a9813b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:26.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9c876700 1 --2- 172.21.15.40:0/1697368288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a980a5fb0 0x7f4a9813b330 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4a90009e90 tx=0x7f4a90009780 comp rx=0 tx=0).stop 2023-09-30T00:50:26.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9c876700 1 -- 172.21.15.40:0/1697368288 shutdown_connections 2023-09-30T00:50:26.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9c876700 1 --2- 172.21.15.40:0/1697368288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a980a5fb0 0x7f4a9813b330 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:26.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9c876700 1 -- 172.21.15.40:0/1697368288 >> 172.21.15.40:0/1697368288 conn(0x7f4a980631f0 msgr2=0x7f4a980635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9c876700 1 -- 172.21.15.40:0/1697368288 shutdown_connections 2023-09-30T00:50:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.871+0000 7f4a9c876700 1 -- 172.21.15.40:0/1697368288 wait complete. 2023-09-30T00:50:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.872+0000 7f4a9c876700 10 monclient: build_initial_monmap 2023-09-30T00:50:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.872+0000 7f4a9c876700 10 monclient: monmap: 2023-09-30T00:50:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:26.879 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.872+0000 7f4a9c876700 1 librados: starting msgr at 2023-09-30T00:50:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.872+0000 7f4a9c876700 1 librados: starting objecter 2023-09-30T00:50:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.872+0000 7f4a9c876700 1 Processor -- start 2023-09-30T00:50:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 1 -- start start 2023-09-30T00:50:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 1 librados: setting wanted keys 2023-09-30T00:50:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 1 librados: calling monclient init 2023-09-30T00:50:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 10 monclient: init 2023-09-30T00:50:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:50:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a980a5fb0 0x7f4a980a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 10 monclient(hunting): picked mon.noname-b con 0x7f4a980a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a980b1460 0x7f4a9813e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 10 monclient(hunting): picked mon.noname-a con 0x7f4a980b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a980b1830 0x7f4a980a9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 10 monclient(hunting): picked mon.noname-c con 0x7f4a980b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980ab880 con 0x7f4a980b1460 2023-09-30T00:50:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980abb60 con 0x7f4a980a5fb0 2023-09-30T00:50:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980a86d0 con 0x7f4a980b1830 2023-09-30T00:50:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.873+0000 7f4a9c876700 10 monclient(hunting): authenticate will time out at 5488.174660s 2023-09-30T00:50:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.874+0000 7f4a96d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a980b1830 0x7f4a980a9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.874+0000 7f4a9659c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a980a5fb0 0x7f4a980a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.874+0000 7f4a96d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a980b1830 0x7f4a980a9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:47698/0 (socket says 172.21.15.40:47698) 2023-09-30T00:50:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.874+0000 7f4a96d9d700 1 -- 172.21.15.40:0/1726130436 learned_addr learned my addr 172.21.15.40:0/1726130436 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.874+0000 7f4a96d9d700 10 monclient(hunting): get_auth_request con 0x7f4a980b1830 auth_method 0 2023-09-30T00:50:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.874+0000 7f4a96d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.874+0000 7f4a96d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.874+0000 7f4a96d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.874+0000 7f4a9659c700 10 monclient(hunting): get_auth_request con 0x7f4a980a5fb0 auth_method 0 2023-09-30T00:50:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.874+0000 7f4a9659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.874+0000 7f4a9659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.874+0000 7f4a9659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.875+0000 7f4a95d9b700 1 --2- 172.21.15.40:0/1726130436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a980b1460 0x7f4a9813e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.875+0000 7f4a96d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.875+0000 7f4a96d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.875+0000 7f4a96d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.875+0000 7f4a9659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.875+0000 7f4a9659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.875+0000 7f4a9659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.875+0000 7f4a95d9b700 10 monclient(hunting): get_auth_request con 0x7f4a980b1460 auth_method 0 2023-09-30T00:50:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.875+0000 7f4a95d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.875+0000 7f4a95d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:26.882 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.875+0000 7f4a95d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.876+0000 7f4a95d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.876+0000 7f4a95d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.876+0000 7f4a95d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.879+0000 7f4a96d9d700 10 monclient(hunting): handle_auth_done global_id 4499 payload 824 2023-09-30T00:50:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.879+0000 7f4a96d9d700 1 -- 172.21.15.40:0/1726130436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a980a5fb0 msgr2=0x7f4a980a7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.879+0000 7f4a96d9d700 1 --2- 172.21.15.40:0/1726130436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a980a5fb0 0x7f4a980a7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.879+0000 7f4a96d9d700 1 -- 172.21.15.40:0/1726130436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a980b1460 msgr2=0x7f4a9813e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.879+0000 7f4a96d9d700 1 --2- 172.21.15.40:0/1726130436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a980b1460 0x7f4a9813e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.879+0000 7f4a96d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:50:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.879+0000 7f4a96d9d700 1 monclient: found mon.noname-c 2023-09-30T00:50:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.879+0000 7f4a96d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.879+0000 7f4a96d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:50:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.879+0000 7f4a96d9d700 1 -- 172.21.15.40:0/1726130436 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a90009a20 con 0x7f4a980b1830 2023-09-30T00:50:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.882+0000 7f4a96d9d700 1 --2- 172.21.15.40:0/1726130436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a980b1830 0x7f4a980a9590 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a900099f0 tx=0x7f4a90018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.882+0000 7f4a8e7fc700 10 client.?.objecter ms_handle_connect 0x7f4a980b1830 2023-09-30T00:50:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.882+0000 7f4a8e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:26.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.886+0000 7f4a8e7fc700 1 -- 172.21.15.40:0/1726130436 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90027470 con 0x7f4a980b1830 2023-09-30T00:50:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.886+0000 7f4a8e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.886+0000 7f4a8e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:50:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.886+0000 7f4a8e7fc700 10 monclient: dump: 2023-09-30T00:50:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:26.889 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.886+0000 7f4a8e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:50:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.886+0000 7f4a8e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a8e7fc700 1 -- 172.21.15.40:0/1726130436 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a900275f0 con 0x7f4a980b1830 2023-09-30T00:50:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a8e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a8e7fc700 1 -- 172.21.15.40:0/1726130436 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a900279b0 con 0x7f4a980b1830 2023-09-30T00:50:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a9c876700 5 monclient: authenticate success, global_id 4499 2023-09-30T00:50:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a9c876700 10 monclient: _renew_subs 2023-09-30T00:50:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a9c876700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a9c876700 1 -- 172.21.15.40:0/1726130436 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9813e730 con 0x7f4a980b1830 2023-09-30T00:50:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a94d99700 4 set_mon_vals no callback set 2023-09-30T00:50:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a8e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a8e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:50:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a8e7fc700 10 monclient: dump: 2023-09-30T00:50:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:26.891 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a9c876700 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a9c876700 10 monclient: _renew_subs 2023-09-30T00:50:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a9c876700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a9c876700 1 -- 172.21.15.40:0/1726130436 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a980a9d10 con 0x7f4a980b1830 2023-09-30T00:50:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.887+0000 7f4a9c876700 1 librados: init done 2023-09-30T00:50:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.888+0000 7f4a9c876700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.888+0000 7f4a9c876700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.888+0000 7f4a9c876700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:26.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.888+0000 7f4a9c876700 1 -- 172.21.15.40:0/1726130436 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a98064a00 con 0x7f4a980b1830 2023-09-30T00:50:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.908+0000 7f4a8e7fc700 1 -- 172.21.15.40:0/1726130436 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f4a90024070 con 0x7f4a980b1830 2023-09-30T00:50:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.908+0000 7f4a8e7fc700 1 --2- 172.21.15.40:0/1726130436 >> v2:172.21.15.135:6800/116426 conn(0x7f4a78068890 0x7f4a7806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.908+0000 7f4a9659c700 1 --2- 172.21.15.40:0/1726130436 >> v2:172.21.15.135:6800/116426 conn(0x7f4a78068890 0x7f4a7806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.908+0000 7f4a9659c700 10 monclient: get_auth_request con 0x7f4a78068890 auth_method 0 2023-09-30T00:50:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.909+0000 7f4a8e7fc700 10 client.4499.objecter ms_handle_connect 0x7f4a78068890 2023-09-30T00:50:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.909+0000 7f4a9659c700 1 --2- 172.21.15.40:0/1726130436 >> v2:172.21.15.135:6800/116426 conn(0x7f4a78068890 0x7f4a7806ad40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4a88000c00 tx=0x7f4a88009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.993+0000 7f4a8e7fc700 1 -- 172.21.15.40:0/1726130436 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f4a900083a0 con 0x7f4a980b1830 2023-09-30T00:50:26.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.993+0000 7f4a8e7fc700 10 client.4499.objecter ms_dispatch 0x7f4a98001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:26.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.993+0000 7f4a8e7fc700 3 client.4499.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:26.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.993+0000 7f4a8e7fc700 3 client.4499.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:26.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:26.994+0000 7f4a8e7fc700 20 client.4499.objecter dump_active .. 0 homeless 2023-09-30T00:50:27.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.181+0000 7f4a8e7fc700 1 -- 172.21.15.40:0/1726130436 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4a90054740 con 0x7f4a980b1830 2023-09-30T00:50:27.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.181+0000 7f4a8e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:27.182 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.181+0000 7f4a8e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:27.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.447+0000 7f4a9c876700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:27.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.447+0000 7f4a9c876700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:27.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.447+0000 7f4a9c876700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:27.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.447+0000 7f4a9c876700 1 -- 172.21.15.40:0/1726130436 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4a980b2f30 con 0x7f4a980b1830 2023-09-30T00:50:27.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.451+0000 7f4a8e7fc700 1 -- 172.21.15.40:0/1726130436 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4a90057d60 con 0x7f4a980b1830 2023-09-30T00:50:27.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.452+0000 7f4a8e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:27.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.452+0000 7f4a8e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:27.452 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:27.452 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:27.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 10 librados: watch_flush enter 2023-09-30T00:50:27.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 10 librados: watch_flush exit 2023-09-30T00:50:27.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 20 client.4499.objecter shutdown clearing up homeless session... 2023-09-30T00:50:27.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 10 client.4499.objecter successfully canceled tick 2023-09-30T00:50:27.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 1 -- 172.21.15.40:0/1726130436 >> v2:172.21.15.135:6800/116426 conn(0x7f4a78068890 msgr2=0x7f4a7806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:27.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 1 --2- 172.21.15.40:0/1726130436 >> v2:172.21.15.135:6800/116426 conn(0x7f4a78068890 0x7f4a7806ad40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4a88000c00 tx=0x7f4a88009040 comp rx=0 tx=0).stop 2023-09-30T00:50:27.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 10 monclient: shutdown 2023-09-30T00:50:27.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:27.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 1 -- 172.21.15.40:0/1726130436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a980b1830 msgr2=0x7f4a980a9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:27.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 1 --2- 172.21.15.40:0/1726130436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a980b1830 0x7f4a980a9590 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4a900099f0 tx=0x7f4a90018000 comp rx=0 tx=0).stop 2023-09-30T00:50:27.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 1 -- 172.21.15.40:0/1726130436 shutdown_connections 2023-09-30T00:50:27.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 1 --2- 172.21.15.40:0/1726130436 >> v2:172.21.15.135:6800/116426 conn(0x7f4a78068890 0x7f4a7806ad40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:27.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 1 --2- 172.21.15.40:0/1726130436 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a980a5fb0 0x7f4a980a7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:27.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 1 --2- 172.21.15.40:0/1726130436 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a980b1830 0x7f4a980a9590 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:27.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 1 --2- 172.21.15.40:0/1726130436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a980b1460 0x7f4a9813e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:27.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 1 -- 172.21.15.40:0/1726130436 >> 172.21.15.40:0/1726130436 conn(0x7f4a980631f0 msgr2=0x7f4a980b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:27.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 1 -- 172.21.15.40:0/1726130436 shutdown_connections 2023-09-30T00:50:27.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 1 -- 172.21.15.40:0/1726130436 wait complete. 2023-09-30T00:50:27.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:27.454+0000 7f4a9c876700 1 librados: shutdown 2023-09-30T00:50:30.470 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:30.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.618+0000 7fcf556b8700 10 monclient: get_monmap_and_config 2023-09-30T00:50:30.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.618+0000 7fcf556b8700 10 monclient: build_initial_monmap 2023-09-30T00:50:30.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.618+0000 7fcf556b8700 10 monclient: monmap: 2023-09-30T00:50:30.619 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:30.619 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:30.619 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:30.619438+0000 2023-09-30T00:50:30.619 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:30.619438+0000 2023-09-30T00:50:30.619 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:30.619 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:30.620 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:30.620 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 1 Processor -- start 2023-09-30T00:50:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 1 -- start start 2023-09-30T00:50:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 10 monclient: init 2023-09-30T00:50:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-09-30T00:50:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf500b1460 0x7fcf500b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 10 monclient(hunting): picked mon.noname-b con 0x7fcf500b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcf500a5fb0 0x7fcf50141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 10 monclient(hunting): picked mon.noname-c con 0x7fcf500a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:50:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcf500af410 0x7fcf500af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 10 monclient(hunting): picked mon.noname-a con 0x7fcf500af410 addr v2:172.21.15.40:3300/0 2023-09-30T00:50:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf500a71f0 con 0x7fcf500af410 2023-09-30T00:50:30.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf500a74d0 con 0x7fcf500a5fb0 2023-09-30T00:50:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf500a63c0 con 0x7fcf500b1460 2023-09-30T00:50:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.619+0000 7fcf556b8700 10 monclient(hunting): authenticate will time out at 5491.920881s 2023-09-30T00:50:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.620+0000 7fcf4e7fc700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcf500a5fb0 0x7fcf50141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.620+0000 7fcf4e7fc700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fcf500a5fb0 0x7fcf50141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:47624/0 (socket says 172.21.15.40:47624) 2023-09-30T00:50:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.620+0000 7fcf4e7fc700 1 -- 172.21.15.40:0/957857444 learned_addr learned my addr 172.21.15.40:0/957857444 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.620+0000 7fcf4e7fc700 10 monclient(hunting): get_auth_request con 0x7fcf500a5fb0 auth_method 0 2023-09-30T00:50:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.620+0000 7fcf4e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.620+0000 7fcf4e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:30.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.620+0000 7fcf4e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.621+0000 7fcf4e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.621+0000 7fcf4e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.621+0000 7fcf4e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.623+0000 7fcf4e7fc700 10 monclient(hunting): handle_auth_done global_id 4621 payload 274 2023-09-30T00:50:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.623+0000 7fcf4e7fc700 1 -- 172.21.15.40:0/957857444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf500b1460 msgr2=0x7fcf500b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:30.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.623+0000 7fcf4e7fc700 1 --2- 172.21.15.40:0/957857444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf500b1460 0x7fcf500b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.623+0000 7fcf4e7fc700 1 -- 172.21.15.40:0/957857444 >> v2:172.21.15.40:3300/0 conn(0x7fcf500af410 msgr2=0x7fcf500af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.623+0000 7fcf4e7fc700 1 --2- 172.21.15.40:0/957857444 >> v2:172.21.15.40:3300/0 conn(0x7fcf500af410 0x7fcf500af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.623+0000 7fcf4e7fc700 10 monclient: _finish_hunting 0 2023-09-30T00:50:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.623+0000 7fcf4e7fc700 1 monclient: found mon.noname-c 2023-09-30T00:50:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.623+0000 7fcf4e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.623+0000 7fcf4e7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:50:30.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.623+0000 7fcf4e7fc700 1 -- 172.21.15.40:0/957857444 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf500a56d0 con 0x7fcf500a5fb0 2023-09-30T00:50:30.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.625+0000 7fcf4e7fc700 1 --2- 172.21.15.40:0/957857444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf500a5fb0 0x7fcf50141730 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcf5006f560 tx=0x7fcf44009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3848c23a006c8fcc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:30.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf4dffb700 1 -- 172.21.15.40:0/957857444 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4400b860 con 0x7fcf500a5fb0 2023-09-30T00:50:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf4dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf4dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:50:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf4dffb700 10 monclient: dump: 2023-09-30T00:50:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:30.632 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf4dffb700 10 monclient: _finish_auth 0 2023-09-30T00:50:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf4dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf4dffb700 1 -- 172.21.15.40:0/957857444 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf4400b9c0 con 0x7fcf500a5fb0 2023-09-30T00:50:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf556b8700 5 monclient: authenticate success, global_id 4621 2023-09-30T00:50:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf556b8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:50:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf4dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf556b8700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:30.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf4dffb700 1 -- 172.21.15.40:0/957857444 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4400bd60 con 0x7fcf500a5fb0 2023-09-30T00:50:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf556b8700 4 set_mon_vals no callback set 2023-09-30T00:50:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf4dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf4dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:50:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf4dffb700 10 monclient: dump: 2023-09-30T00:50:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:30.633 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:30.634 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf556b8700 10 monclient: shutdown 2023-09-30T00:50:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf556b8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf556b8700 1 -- 172.21.15.40:0/957857444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf500a5fb0 msgr2=0x7fcf50141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.630+0000 7fcf556b8700 1 --2- 172.21.15.40:0/957857444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf500a5fb0 0x7fcf50141730 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcf5006f560 tx=0x7fcf44009250 comp rx=0 tx=0).stop 2023-09-30T00:50:30.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.631+0000 7fcf556b8700 1 -- 172.21.15.40:0/957857444 shutdown_connections 2023-09-30T00:50:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.631+0000 7fcf556b8700 1 --2- 172.21.15.40:0/957857444 >> v2:172.21.15.40:3300/0 conn(0x7fcf500af410 0x7fcf500af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.631+0000 7fcf556b8700 1 --2- 172.21.15.40:0/957857444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf500b1460 0x7fcf500b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.631+0000 7fcf556b8700 1 --2- 172.21.15.40:0/957857444 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf500a5fb0 0x7fcf50141730 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.631+0000 7fcf556b8700 1 -- 172.21.15.40:0/957857444 >> 172.21.15.40:0/957857444 conn(0x7fcf500631f0 msgr2=0x7fcf500635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.631+0000 7fcf556b8700 1 -- 172.21.15.40:0/957857444 shutdown_connections 2023-09-30T00:50:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.631+0000 7fcf556b8700 1 -- 172.21.15.40:0/957857444 wait complete. 2023-09-30T00:50:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.632+0000 7fcf556b8700 10 monclient: build_initial_monmap 2023-09-30T00:50:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.632+0000 7fcf556b8700 10 monclient: monmap: 2023-09-30T00:50:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:30.637 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:30.638 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.632+0000 7fcf556b8700 1 librados: starting msgr at 2023-09-30T00:50:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.632+0000 7fcf556b8700 1 librados: starting objecter 2023-09-30T00:50:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.632+0000 7fcf556b8700 1 Processor -- start 2023-09-30T00:50:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.632+0000 7fcf556b8700 1 -- start start 2023-09-30T00:50:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.632+0000 7fcf556b8700 1 librados: setting wanted keys 2023-09-30T00:50:30.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.632+0000 7fcf556b8700 1 librados: calling monclient init 2023-09-30T00:50:30.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.632+0000 7fcf556b8700 10 monclient: init 2023-09-30T00:50:30.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:30.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:50:30.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf500a5fb0 0x7fcf500a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:30.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 10 monclient(hunting): picked mon.noname-c con 0x7fcf500a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:30.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf500af410 0x7fcf500a8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:30.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 10 monclient(hunting): picked mon.noname-b con 0x7fcf500af410 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:30.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf500b1460 0x7fcf5013cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:30.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 10 monclient(hunting): picked mon.noname-a con 0x7fcf500b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:30.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:30.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf500aa640 con 0x7fcf500b1460 2023-09-30T00:50:30.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:30.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf500a8d60 con 0x7fcf500af410 2023-09-30T00:50:30.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:30.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf500a9530 con 0x7fcf500a5fb0 2023-09-30T00:50:30.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:30.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf556b8700 10 monclient(hunting): authenticate will time out at 5491.934218s 2023-09-30T00:50:30.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.633+0000 7fcf4effd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf500a5fb0 0x7fcf500a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:30.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.634+0000 7fcf4e7fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf500af410 0x7fcf500a8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:30.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.634+0000 7fcf4effd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf500a5fb0 0x7fcf500a81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:47730/0 (socket says 172.21.15.40:47730) 2023-09-30T00:50:30.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.634+0000 7fcf4effd700 1 -- 172.21.15.40:0/598874032 learned_addr learned my addr 172.21.15.40:0/598874032 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:30.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.634+0000 7fcf4effd700 10 monclient(hunting): get_auth_request con 0x7fcf500a5fb0 auth_method 0 2023-09-30T00:50:30.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.634+0000 7fcf4effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:30.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.634+0000 7fcf4effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:30.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.634+0000 7fcf4effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:30.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.634+0000 7fcf4e7fc700 10 monclient(hunting): get_auth_request con 0x7fcf500af410 auth_method 0 2023-09-30T00:50:30.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.634+0000 7fcf4e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:30.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.634+0000 7fcf4e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:30.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.634+0000 7fcf4e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:30.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.634+0000 7fcf4effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:30.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.634+0000 7fcf4effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:30.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.634+0000 7fcf4effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:30.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.635+0000 7fcf4e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:30.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.635+0000 7fcf4e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:30.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.635+0000 7fcf4e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:30.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.639+0000 7fcf4e7fc700 10 monclient(hunting): handle_auth_done global_id 4624 payload 824 2023-09-30T00:50:30.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.639+0000 7fcf4e7fc700 1 -- 172.21.15.40:0/598874032 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf500a5fb0 msgr2=0x7fcf500a81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:30.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.639+0000 7fcf4e7fc700 1 --2- 172.21.15.40:0/598874032 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf500a5fb0 0x7fcf500a81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:30.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.639+0000 7fcf4e7fc700 1 -- 172.21.15.40:0/598874032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf500b1460 msgr2=0x7fcf5013cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:30.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.639+0000 7fcf4e7fc700 1 --2- 172.21.15.40:0/598874032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf500b1460 0x7fcf5013cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:30.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.639+0000 7fcf4e7fc700 10 monclient: _finish_hunting 0 2023-09-30T00:50:30.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.639+0000 7fcf4e7fc700 1 monclient: found mon.noname-b 2023-09-30T00:50:30.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.639+0000 7fcf4e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:30.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.639+0000 7fcf4e7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:50:30.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.639+0000 7fcf4e7fc700 1 -- 172.21.15.40:0/598874032 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf440094f0 con 0x7fcf500af410 2023-09-30T00:50:30.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.642+0000 7fcf4e7fc700 1 --2- 172.21.15.40:0/598874032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf500af410 0x7fcf500a8820 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcf500b05f0 tx=0x7fcf44017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:30.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.642+0000 7fcf36ffd700 10 client.?.objecter ms_handle_connect 0x7fcf500af410 2023-09-30T00:50:30.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.642+0000 7fcf36ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:30.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf36ffd700 1 -- 172.21.15.40:0/598874032 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4401a450 con 0x7fcf500af410 2023-09-30T00:50:30.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf36ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:30.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf36ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:30.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf36ffd700 10 monclient: dump: 2023-09-30T00:50:30.649 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:30.649 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:30.649 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:30.650 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf36ffd700 10 monclient: _finish_auth 0 2023-09-30T00:50:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf36ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf36ffd700 1 -- 172.21.15.40:0/598874032 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf4401a5d0 con 0x7fcf500af410 2023-09-30T00:50:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf556b8700 5 monclient: authenticate success, global_id 4624 2023-09-30T00:50:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf556b8700 10 monclient: _renew_subs 2023-09-30T00:50:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf556b8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf556b8700 1 -- 172.21.15.40:0/598874032 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf500a90a0 con 0x7fcf500af410 2023-09-30T00:50:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf36ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf36ffd700 1 -- 172.21.15.40:0/598874032 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4401a990 con 0x7fcf500af410 2023-09-30T00:50:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf36ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf36ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:50:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf36ffd700 10 monclient: dump: 2023-09-30T00:50:30.652 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:30.652 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:30.652 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:30.652 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:30.652 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:30.652 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:30.652 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:30.652 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:30.652 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:30.652 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:30.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf4d7fa700 4 set_mon_vals no callback set 2023-09-30T00:50:30.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf556b8700 10 client.4624.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf556b8700 10 monclient: _renew_subs 2023-09-30T00:50:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf556b8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf556b8700 1 -- 172.21.15.40:0/598874032 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5013d4c0 con 0x7fcf500af410 2023-09-30T00:50:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.648+0000 7fcf556b8700 1 librados: init done 2023-09-30T00:50:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.649+0000 7fcf556b8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.649+0000 7fcf556b8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:30.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.649+0000 7fcf556b8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:30.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.649+0000 7fcf556b8700 1 -- 172.21.15.40:0/598874032 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf500b0fe0 con 0x7fcf500af410 2023-09-30T00:50:30.668 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.668+0000 7fcf36ffd700 1 -- 172.21.15.40:0/598874032 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fcf4401ad40 con 0x7fcf500af410 2023-09-30T00:50:30.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.668+0000 7fcf36ffd700 1 --2- 172.21.15.40:0/598874032 >> v2:172.21.15.135:6800/116426 conn(0x7fcf3c068960 0x7fcf3c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:30.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.668+0000 7fcf4effd700 1 --2- 172.21.15.40:0/598874032 >> v2:172.21.15.135:6800/116426 conn(0x7fcf3c068960 0x7fcf3c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:30.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.668+0000 7fcf4effd700 10 monclient: get_auth_request con 0x7fcf3c068960 auth_method 0 2023-09-30T00:50:30.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.669+0000 7fcf4effd700 1 --2- 172.21.15.40:0/598874032 >> v2:172.21.15.135:6800/116426 conn(0x7fcf3c068960 0x7fcf3c06ae10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcf40009b80 tx=0x7fcf40006f90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:30.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.669+0000 7fcf36ffd700 10 client.4624.objecter ms_handle_connect 0x7fcf3c068960 2023-09-30T00:50:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.901+0000 7fcf36ffd700 1 -- 172.21.15.40:0/598874032 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fcf44057600 con 0x7fcf500af410 2023-09-30T00:50:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.901+0000 7fcf36ffd700 10 client.4624.objecter ms_dispatch 0x7fcf50001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.901+0000 7fcf36ffd700 3 client.4624.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.901+0000 7fcf36ffd700 3 client.4624.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.901+0000 7fcf36ffd700 20 client.4624.objecter dump_active .. 0 homeless 2023-09-30T00:50:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.910+0000 7fcf36ffd700 1 -- 172.21.15.40:0/598874032 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcf44053240 con 0x7fcf500af410 2023-09-30T00:50:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.910+0000 7fcf36ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:30.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:30.910+0000 7fcf36ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:31.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.162+0000 7fcf556b8700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:31.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.162+0000 7fcf556b8700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:31.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.162+0000 7fcf556b8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:31.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.162+0000 7fcf556b8700 1 -- 172.21.15.40:0/598874032 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcf500a8ef0 con 0x7fcf500af410 2023-09-30T00:50:31.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.168+0000 7fcf36ffd700 1 -- 172.21.15.40:0/598874032 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcf4402a100 con 0x7fcf500af410 2023-09-30T00:50:31.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.168+0000 7fcf36ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:31.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.168+0000 7fcf36ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:31.169 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:31.169 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 10 librados: watch_flush enter 2023-09-30T00:50:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 10 librados: watch_flush exit 2023-09-30T00:50:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 20 client.4624.objecter shutdown clearing up homeless session... 2023-09-30T00:50:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 10 client.4624.objecter successfully canceled tick 2023-09-30T00:50:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 1 -- 172.21.15.40:0/598874032 >> v2:172.21.15.135:6800/116426 conn(0x7fcf3c068960 msgr2=0x7fcf3c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 1 --2- 172.21.15.40:0/598874032 >> v2:172.21.15.135:6800/116426 conn(0x7fcf3c068960 0x7fcf3c06ae10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fcf40009b80 tx=0x7fcf40006f90 comp rx=0 tx=0).stop 2023-09-30T00:50:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 10 monclient: shutdown 2023-09-30T00:50:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 1 -- 172.21.15.40:0/598874032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf500af410 msgr2=0x7fcf500a8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 1 --2- 172.21.15.40:0/598874032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf500af410 0x7fcf500a8820 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcf500b05f0 tx=0x7fcf44017000 comp rx=0 tx=0).stop 2023-09-30T00:50:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 1 -- 172.21.15.40:0/598874032 shutdown_connections 2023-09-30T00:50:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 1 --2- 172.21.15.40:0/598874032 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf500b1460 0x7fcf5013cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 1 --2- 172.21.15.40:0/598874032 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fcf500a5fb0 0x7fcf500a81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:31.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 1 --2- 172.21.15.40:0/598874032 >> v2:172.21.15.135:6800/116426 conn(0x7fcf3c068960 0x7fcf3c06ae10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:31.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 1 --2- 172.21.15.40:0/598874032 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fcf500af410 0x7fcf500a8820 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:31.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 1 -- 172.21.15.40:0/598874032 >> 172.21.15.40:0/598874032 conn(0x7fcf500631f0 msgr2=0x7fcf501399c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:31.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 1 -- 172.21.15.40:0/598874032 shutdown_connections 2023-09-30T00:50:31.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.170+0000 7fcf556b8700 1 -- 172.21.15.40:0/598874032 wait complete. 2023-09-30T00:50:31.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:31.171+0000 7fcf556b8700 1 librados: shutdown 2023-09-30T00:50:33.306 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:04:09.017 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:33.310 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:04:09.021 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:33.342 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:04:09.053 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:33.362 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:04:09.049 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:33.366 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:04:09.053 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:33.368 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:04:09.080 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:33.372 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:04:09.084 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:33.392 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:04:09.079 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:33.403 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:04:09.115 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:50:34.185 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:34.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.330+0000 7f6b0b428700 10 monclient: get_monmap_and_config 2023-09-30T00:50:34.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.330+0000 7f6b0b428700 10 monclient: build_initial_monmap 2023-09-30T00:50:34.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.330+0000 7f6b0b428700 10 monclient: monmap: 2023-09-30T00:50:34.331 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:34.331 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:34.331 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:34.331671+0000 2023-09-30T00:50:34.331 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:34.331671+0000 2023-09-30T00:50:34.331 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:34.331 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:34.332 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:34.332 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:34.332 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:34.332 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:34.332 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:34.332 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:34.333 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.331+0000 7f6b0b428700 1 Processor -- start 2023-09-30T00:50:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.331+0000 7f6b0b428700 1 -- start start 2023-09-30T00:50:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.331+0000 7f6b0b428700 10 monclient: init 2023-09-30T00:50:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b0b428700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b0b428700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-09-30T00:50:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b0b428700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6b040b1460 0x7f6b040b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b0b428700 10 monclient(hunting): picked mon.noname-c con 0x7f6b040b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:50:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b0b428700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b040a5fb0 0x7f6b04141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b0b428700 10 monclient(hunting): picked mon.noname-b con 0x7f6b040a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b0b428700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b040af410 addr v1:172.21.15.40:6789/0 2023-09-30T00:50:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b0b428700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b040a71f0 con 0x7f6b040af410 2023-09-30T00:50:34.334 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b0b428700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b0b428700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b040a74d0 con 0x7f6b040b1460 2023-09-30T00:50:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b0b428700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b0b428700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b040a63c0 con 0x7f6b040a5fb0 2023-09-30T00:50:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b0b428700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b0b428700 10 monclient(hunting): authenticate will time out at 5495.633312s 2023-09-30T00:50:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b099c5700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6b040af410 0x7f6b040af890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48372/0 (socket says 172.21.15.40:48372) 2023-09-30T00:50:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b099c5700 1 -- 172.21.15.40:0/1896103330 learned_addr learned my addr 172.21.15.40:0/1896103330 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:50:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.332+0000 7f6b089c3700 1 --2- 172.21.15.40:0/1896103330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b040a5fb0 0x7f6b04141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.333+0000 7f6b091c4700 1 --2- 172.21.15.40:0/1896103330 >> v2:172.21.15.135:3300/0 conn(0x7f6b040b1460 0x7f6b040b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.333+0000 7f6b089c3700 10 monclient(hunting): get_auth_request con 0x7f6b040a5fb0 auth_method 0 2023-09-30T00:50:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.333+0000 7f6b089c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:34.335 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.333+0000 7f6b089c3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:34.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.333+0000 7f6b089c3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:34.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.333+0000 7f6b091c4700 10 monclient(hunting): get_auth_request con 0x7f6b040b1460 auth_method 0 2023-09-30T00:50:34.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.333+0000 7f6b091c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:34.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.333+0000 7f6b091c4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:34.336 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.333+0000 7f6b091c4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.334+0000 7f6b089c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.334+0000 7f6b089c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.334+0000 7f6b089c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.334+0000 7f6b091c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.334+0000 7f6b091c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.334+0000 7f6b091c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.336+0000 7f6afbfff700 1 -- 172.21.15.40:0/1896103330 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158359028 0 0) 0x7f6b040a71f0 con 0x7f6b040af410 2023-09-30T00:50:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.336+0000 7f6afbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.336+0000 7f6afbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.336+0000 7f6afbfff700 10 monclient(hunting): my global_id is 4563 2023-09-30T00:50:34.337 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.336+0000 7f6afbfff700 1 -- 172.21.15.40:0/1896103330 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6af0003570 con 0x7f6b040af410 2023-09-30T00:50:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.337+0000 7f6b091c4700 10 monclient(hunting): handle_auth_done global_id 4627 payload 274 2023-09-30T00:50:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.337+0000 7f6b091c4700 1 -- 172.21.15.40:0/1896103330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b040a5fb0 msgr2=0x7f6b04141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.337+0000 7f6b091c4700 1 --2- 172.21.15.40:0/1896103330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b040a5fb0 0x7f6b04141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.337+0000 7f6b091c4700 1 -- 172.21.15.40:0/1896103330 >> v1:172.21.15.40:6789/0 conn(0x7f6b040af410 legacy=0x7f6b040af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:34.338 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.337+0000 7f6b091c4700 10 monclient: _finish_hunting 0 2023-09-30T00:50:34.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.337+0000 7f6b091c4700 1 monclient: found mon.noname-c 2023-09-30T00:50:34.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.337+0000 7f6b091c4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:34.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.337+0000 7f6b091c4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:50:34.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.337+0000 7f6b091c4700 1 -- 172.21.15.40:0/1896103330 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b040a56d0 con 0x7f6b040b1460 2023-09-30T00:50:34.339 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.339+0000 7f6b091c4700 1 --2- 172.21.15.40:0/1896103330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b040b1460 0x7f6b040b1830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6af40098e0 tx=0x7f6af40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcd24cba72111068 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:34.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.343+0000 7f6afbfff700 1 -- 172.21.15.40:0/1896103330 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4011630 con 0x7f6b040b1460 2023-09-30T00:50:34.345 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.343+0000 7f6afbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:34.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6afbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:50:34.346 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6afbfff700 10 monclient: dump: 2023-09-30T00:50:34.346 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:34.346 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:34.346 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:34.346 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:34.346 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:34.346 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:34.346 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:34.347 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:34.347 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:34.347 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:34.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6afbfff700 10 monclient: _finish_auth 0 2023-09-30T00:50:34.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6afbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:34.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6afbfff700 1 -- 172.21.15.40:0/1896103330 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af4011790 con 0x7f6b040b1460 2023-09-30T00:50:34.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6b0b428700 5 monclient: authenticate success, global_id 4627 2023-09-30T00:50:34.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6b0b428700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:50:34.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6afbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:34.347 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6b0b428700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:34.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6b0b428700 4 set_mon_vals no callback set 2023-09-30T00:50:34.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6afbfff700 1 -- 172.21.15.40:0/1896103330 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4011b30 con 0x7f6b040b1460 2023-09-30T00:50:34.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6afbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:34.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6afbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:50:34.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6afbfff700 10 monclient: dump: 2023-09-30T00:50:34.348 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:34.348 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:34.348 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:34.348 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:34.349 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:34.349 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:34.349 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:34.349 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:34.349 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:34.349 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:34.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6b0b428700 10 monclient: shutdown 2023-09-30T00:50:34.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6b0b428700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:34.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6b0b428700 1 -- 172.21.15.40:0/1896103330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b040b1460 msgr2=0x7f6b040b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:34.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6b0b428700 1 --2- 172.21.15.40:0/1896103330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b040b1460 0x7f6b040b1830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6af40098e0 tx=0x7f6af40091a0 comp rx=0 tx=0).stop 2023-09-30T00:50:34.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6b0b428700 1 -- 172.21.15.40:0/1896103330 shutdown_connections 2023-09-30T00:50:34.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6b0b428700 1 --2- 172.21.15.40:0/1896103330 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b040b1460 0x7f6b040b1830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:34.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6b0b428700 1 --2- 172.21.15.40:0/1896103330 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b040a5fb0 0x7f6b04141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:34.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6b0b428700 1 -- 172.21.15.40:0/1896103330 >> 172.21.15.40:0/1896103330 conn(0x7f6b040631f0 msgr2=0x7f6b040635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:34.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.344+0000 7f6b0b428700 1 -- 172.21.15.40:0/1896103330 shutdown_connections 2023-09-30T00:50:34.350 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.345+0000 7f6b0b428700 1 -- 172.21.15.40:0/1896103330 wait complete. 2023-09-30T00:50:34.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.345+0000 7f6b0b428700 10 monclient: build_initial_monmap 2023-09-30T00:50:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.345+0000 7f6b0b428700 10 monclient: monmap: 2023-09-30T00:50:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:34.354 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:34.355 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.346+0000 7f6b0b428700 1 librados: starting msgr at 2023-09-30T00:50:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.346+0000 7f6b0b428700 1 librados: starting objecter 2023-09-30T00:50:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.346+0000 7f6b0b428700 1 Processor -- start 2023-09-30T00:50:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.346+0000 7f6b0b428700 1 -- start start 2023-09-30T00:50:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.346+0000 7f6b0b428700 1 librados: setting wanted keys 2023-09-30T00:50:34.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.346+0000 7f6b0b428700 1 librados: calling monclient init 2023-09-30T00:50:34.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.346+0000 7f6b0b428700 10 monclient: init 2023-09-30T00:50:34.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:34.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:50:34.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b040a5fb0 0x7f6b0413ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:34.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 10 monclient(hunting): picked mon.noname-b con 0x7f6b040a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:34.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b040af410 0x7f6b041558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:34.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 10 monclient(hunting): picked mon.noname-a con 0x7f6b040af410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:34.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b040b1460 0x7f6b04155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:34.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 10 monclient(hunting): picked mon.noname-c con 0x7f6b040b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:34.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:34.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b04155450 con 0x7f6b040af410 2023-09-30T00:50:34.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:34.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0406ea90 con 0x7f6b040a5fb0 2023-09-30T00:50:34.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:34.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0413d150 con 0x7f6b040b1460 2023-09-30T00:50:34.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:34.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.347+0000 7f6b0b428700 10 monclient(hunting): authenticate will time out at 5495.648626s 2023-09-30T00:50:34.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b091c4700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b040a5fb0 0x7f6b0413ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:34.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b089c3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b040af410 0x7f6b041558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:34.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b099c5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b040b1460 0x7f6b04155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:34.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b091c4700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b040a5fb0 0x7f6b0413ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:51320/0 (socket says 172.21.15.40:51320) 2023-09-30T00:50:34.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b091c4700 1 -- 172.21.15.40:0/2675003517 learned_addr learned my addr 172.21.15.40:0/2675003517 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:34.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b091c4700 10 monclient(hunting): get_auth_request con 0x7f6b040a5fb0 auth_method 0 2023-09-30T00:50:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b091c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b091c4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b091c4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b099c5700 10 monclient(hunting): get_auth_request con 0x7f6b040b1460 auth_method 0 2023-09-30T00:50:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b099c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b099c5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b099c5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b089c3700 10 monclient(hunting): get_auth_request con 0x7f6b040af410 auth_method 0 2023-09-30T00:50:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b089c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:34.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b089c3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.348+0000 7f6b089c3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.349+0000 7f6b091c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.349+0000 7f6b091c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.349+0000 7f6b091c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.349+0000 7f6b099c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.349+0000 7f6b099c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.349+0000 7f6b099c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.349+0000 7f6b089c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.349+0000 7f6b089c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.349+0000 7f6b089c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:34.360 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.352+0000 7f6b091c4700 10 monclient(hunting): handle_auth_done global_id 4630 payload 824 2023-09-30T00:50:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.352+0000 7f6b091c4700 1 -- 172.21.15.40:0/2675003517 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b040b1460 msgr2=0x7f6b04155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.352+0000 7f6b091c4700 1 --2- 172.21.15.40:0/2675003517 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b040b1460 0x7f6b04155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.352+0000 7f6b091c4700 1 -- 172.21.15.40:0/2675003517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b040af410 msgr2=0x7f6b041558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.352+0000 7f6b091c4700 1 --2- 172.21.15.40:0/2675003517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b040af410 0x7f6b041558e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.352+0000 7f6b091c4700 10 monclient: _finish_hunting 0 2023-09-30T00:50:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.352+0000 7f6b091c4700 1 monclient: found mon.noname-b 2023-09-30T00:50:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.352+0000 7f6b091c4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.352+0000 7f6b091c4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:50:34.361 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.352+0000 7f6b091c4700 1 -- 172.21.15.40:0/2675003517 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af4009420 con 0x7f6b040a5fb0 2023-09-30T00:50:34.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.354+0000 7f6b091c4700 1 --2- 172.21.15.40:0/2675003517 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b040a5fb0 0x7f6b0413ded0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6b0406f5e0 tx=0x7f6af4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:34.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.354+0000 7f6af97fa700 10 client.?.objecter ms_handle_connect 0x7f6b040a5fb0 2023-09-30T00:50:34.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.354+0000 7f6af97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:34.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6af97fa700 1 -- 172.21.15.40:0/2675003517 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4000d10 con 0x7f6b040a5fb0 2023-09-30T00:50:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6af97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6af97fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6af97fa700 10 monclient: dump: 2023-09-30T00:50:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:34.364 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:34.365 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6af97fa700 10 monclient: _finish_auth 0 2023-09-30T00:50:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6af97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6af97fa700 1 -- 172.21.15.40:0/2675003517 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af4011b10 con 0x7f6b040a5fb0 2023-09-30T00:50:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6b0b428700 5 monclient: authenticate success, global_id 4630 2023-09-30T00:50:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6af97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6af97fa700 1 -- 172.21.15.40:0/2675003517 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af4025620 con 0x7f6b040a5fb0 2023-09-30T00:50:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6b0b428700 10 monclient: _renew_subs 2023-09-30T00:50:34.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6b0b428700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:34.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6b0b428700 1 -- 172.21.15.40:0/2675003517 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b0413c510 con 0x7f6b040a5fb0 2023-09-30T00:50:34.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6af97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:34.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6afb7fe700 4 set_mon_vals no callback set 2023-09-30T00:50:34.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6af97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:50:34.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6af97fa700 10 monclient: dump: 2023-09-30T00:50:34.366 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:34.366 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:34.366 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:34.367 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6b0b428700 10 client.4630.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6b0b428700 10 monclient: _renew_subs 2023-09-30T00:50:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6b0b428700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:34.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6b0b428700 1 -- 172.21.15.40:0/2675003517 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b04137a50 con 0x7f6b040a5fb0 2023-09-30T00:50:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.358+0000 7f6b0b428700 1 librados: init done 2023-09-30T00:50:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.359+0000 7f6b0b428700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.359+0000 7f6b0b428700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.359+0000 7f6b0b428700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:34.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.359+0000 7f6b0b428700 1 -- 172.21.15.40:0/2675003517 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b04155730 con 0x7f6b040a5fb0 2023-09-30T00:50:34.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.531+0000 7f6af97fa700 1 -- 172.21.15.40:0/2675003517 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6af40259d0 con 0x7f6b040a5fb0 2023-09-30T00:50:34.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.531+0000 7f6af97fa700 1 --2- 172.21.15.40:0/2675003517 >> v2:172.21.15.135:6800/116426 conn(0x7f6ae8068960 0x7f6ae806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:34.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.532+0000 7f6b089c3700 1 --2- 172.21.15.40:0/2675003517 >> v2:172.21.15.135:6800/116426 conn(0x7f6ae8068960 0x7f6ae806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:34.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.532+0000 7f6b089c3700 10 monclient: get_auth_request con 0x7f6ae8068960 auth_method 0 2023-09-30T00:50:34.532 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.532+0000 7f6b089c3700 1 --2- 172.21.15.40:0/2675003517 >> v2:172.21.15.135:6800/116426 conn(0x7f6ae8068960 0x7f6ae806ae10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6b041448b0 tx=0x7f6b000091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:34.533 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.532+0000 7f6af97fa700 10 client.4630.objecter ms_handle_connect 0x7f6ae8068960 2023-09-30T00:50:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.546+0000 7f6af97fa700 1 -- 172.21.15.40:0/2675003517 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6af4057990 con 0x7f6b040a5fb0 2023-09-30T00:50:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.546+0000 7f6af97fa700 10 client.4630.objecter ms_dispatch 0x7f6b04001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.546+0000 7f6af97fa700 3 client.4630.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.546+0000 7f6af97fa700 3 client.4630.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:34.546 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.546+0000 7f6af97fa700 20 client.4630.objecter dump_active .. 0 homeless 2023-09-30T00:50:34.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.630+0000 7f6af97fa700 1 -- 172.21.15.40:0/2675003517 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6af40535d0 con 0x7f6b040a5fb0 2023-09-30T00:50:34.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.630+0000 7f6af97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:34.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.630+0000 7f6af97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:34.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.892+0000 7f6b0b428700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:34.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.892+0000 7f6b0b428700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:34.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.892+0000 7f6b0b428700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:34.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.892+0000 7f6b0b428700 1 -- 172.21.15.40:0/2675003517 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6b04064a00 con 0x7f6b040a5fb0 2023-09-30T00:50:34.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.898+0000 7f6af97fa700 1 -- 172.21.15.40:0/2675003517 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6af4033100 con 0x7f6b040a5fb0 2023-09-30T00:50:34.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.898+0000 7f6af97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:34.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.898+0000 7f6af97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:34.899 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:34.899 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 10 librados: watch_flush enter 2023-09-30T00:50:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 10 librados: watch_flush exit 2023-09-30T00:50:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 20 client.4630.objecter shutdown clearing up homeless session... 2023-09-30T00:50:34.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 10 client.4630.objecter successfully canceled tick 2023-09-30T00:50:34.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 1 -- 172.21.15.40:0/2675003517 >> v2:172.21.15.135:6800/116426 conn(0x7f6ae8068960 msgr2=0x7f6ae806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:34.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 1 --2- 172.21.15.40:0/2675003517 >> v2:172.21.15.135:6800/116426 conn(0x7f6ae8068960 0x7f6ae806ae10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6b041448b0 tx=0x7f6b000091a0 comp rx=0 tx=0).stop 2023-09-30T00:50:34.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 10 monclient: shutdown 2023-09-30T00:50:34.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:34.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 1 -- 172.21.15.40:0/2675003517 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b040a5fb0 msgr2=0x7f6b0413ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:34.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 1 --2- 172.21.15.40:0/2675003517 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b040a5fb0 0x7f6b0413ded0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6b0406f5e0 tx=0x7f6af4017000 comp rx=0 tx=0).stop 2023-09-30T00:50:34.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 1 -- 172.21.15.40:0/2675003517 shutdown_connections 2023-09-30T00:50:34.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 1 --2- 172.21.15.40:0/2675003517 >> v2:172.21.15.135:6800/116426 conn(0x7f6ae8068960 0x7f6ae806ae10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:34.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 1 --2- 172.21.15.40:0/2675003517 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6b040a5fb0 0x7f6b0413ded0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:34.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 1 --2- 172.21.15.40:0/2675003517 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6b040b1460 0x7f6b04155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:34.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 1 --2- 172.21.15.40:0/2675003517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6b040af410 0x7f6b041558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:34.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 1 -- 172.21.15.40:0/2675003517 >> 172.21.15.40:0/2675003517 conn(0x7f6b040631f0 msgr2=0x7f6b0413b370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:34.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 1 -- 172.21.15.40:0/2675003517 shutdown_connections 2023-09-30T00:50:34.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 1 -- 172.21.15.40:0/2675003517 wait complete. 2023-09-30T00:50:34.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:34.900+0000 7f6b0b428700 1 librados: shutdown 2023-09-30T00:50:37.916 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:38.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.055+0000 7f56a3b05700 10 monclient: get_monmap_and_config 2023-09-30T00:50:38.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.055+0000 7f56a3b05700 10 monclient: build_initial_monmap 2023-09-30T00:50:38.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.055+0000 7f56a3b05700 10 monclient: monmap: 2023-09-30T00:50:38.056 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:38.056 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:38.056 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:38.056209+0000 2023-09-30T00:50:38.056 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:38.056209+0000 2023-09-30T00:50:38.056 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:38.056 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:38.056 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:38.056 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:38.056 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:38.056 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:38.057 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:38.057 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:38.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.055+0000 7f56a3b05700 1 Processor -- start 2023-09-30T00:50:38.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 1 -- start start 2023-09-30T00:50:38.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 10 monclient: init 2023-09-30T00:50:38.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:38.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-09-30T00:50:38.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f569c0b1460 0x7f569c0b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 10 monclient(hunting): picked mon.noname-b con 0x7f569c0b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f569c0a5fb0 0x7f569c141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 10 monclient(hunting): picked mon.noname-a con 0x7f569c0a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:50:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f569c0af410 addr v1:172.21.15.135:6789/0 2023-09-30T00:50:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569c0a71f0 con 0x7f569c0af410 2023-09-30T00:50:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f569c0a74d0 con 0x7f569c0a5fb0 2023-09-30T00:50:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569c0a63c0 con 0x7f569c0b1460 2023-09-30T00:50:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.056+0000 7f56a3b05700 10 monclient(hunting): authenticate will time out at 5499.357876s 2023-09-30T00:50:38.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.057+0000 7f56a10a0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f569c0a5fb0 0x7f569c141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.057+0000 7f56a10a0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f569c0a5fb0 0x7f569c141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42778/0 (socket says 172.21.15.40:42778) 2023-09-30T00:50:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.057+0000 7f56a10a0700 1 -- 172.21.15.40:0/764939955 learned_addr learned my addr 172.21.15.40:0/764939955 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.057+0000 7f56a10a0700 10 monclient(hunting): get_auth_request con 0x7f569c0a5fb0 auth_method 0 2023-09-30T00:50:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.057+0000 7f56a10a0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.057+0000 7f56a10a0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.057+0000 7f56a10a0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.058+0000 7f56a10a0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.058+0000 7f56a10a0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:38.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.058+0000 7f56a10a0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.061+0000 7f56a10a0700 10 monclient(hunting): handle_auth_done global_id 4569 payload 274 2023-09-30T00:50:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.061+0000 7f56a10a0700 1 -- 172.21.15.40:0/764939955 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f569c0b1460 msgr2=0x7f569c0b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.061+0000 7f56a10a0700 1 --2- 172.21.15.40:0/764939955 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f569c0b1460 0x7f569c0b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.061+0000 7f56a10a0700 1 -- 172.21.15.40:0/764939955 >> v1:172.21.15.135:6789/0 conn(0x7f569c0af410 legacy=0x7f569c0af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:38.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.061+0000 7f56a10a0700 10 monclient: _finish_hunting 0 2023-09-30T00:50:38.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.061+0000 7f56a10a0700 1 monclient: found mon.noname-a 2023-09-30T00:50:38.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.061+0000 7f56a10a0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:38.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.061+0000 7f56a10a0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:50:38.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.061+0000 7f56a10a0700 1 -- 172.21.15.40:0/764939955 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569c0a56d0 con 0x7f569c0a5fb0 2023-09-30T00:50:38.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.063+0000 7f56a10a0700 1 --2- 172.21.15.40:0/764939955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f569c0a5fb0 0x7f569c141730 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f568c0099b0 tx=0x7f568c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40dfd23e4949c910 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a089f700 1 -- 172.21.15.40:0/764939955 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c00b820 con 0x7f569c0a5fb0 2023-09-30T00:50:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a089f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a089f700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:50:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a089f700 10 monclient: dump: 2023-09-30T00:50:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:38.069 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:38.070 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a089f700 10 monclient: _finish_auth 0 2023-09-30T00:50:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a089f700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a089f700 1 -- 172.21.15.40:0/764939955 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568c00b980 con 0x7f569c0a5fb0 2023-09-30T00:50:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a3b05700 5 monclient: authenticate success, global_id 4569 2023-09-30T00:50:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a3b05700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:50:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a089f700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a089f700 1 -- 172.21.15.40:0/764939955 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c00bd20 con 0x7f569c0a5fb0 2023-09-30T00:50:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a3b05700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:38.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a3b05700 4 set_mon_vals no callback set 2023-09-30T00:50:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a089f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a089f700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:50:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.068+0000 7f56a089f700 10 monclient: dump: 2023-09-30T00:50:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:38.071 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:38.072 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.069+0000 7f56a3b05700 10 monclient: shutdown 2023-09-30T00:50:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.069+0000 7f56a3b05700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.069+0000 7f56a3b05700 1 -- 172.21.15.40:0/764939955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f569c0a5fb0 msgr2=0x7f569c141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.069+0000 7f56a3b05700 1 --2- 172.21.15.40:0/764939955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f569c0a5fb0 0x7f569c141730 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f568c0099b0 tx=0x7f568c009250 comp rx=0 tx=0).stop 2023-09-30T00:50:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.069+0000 7f56a3b05700 1 -- 172.21.15.40:0/764939955 shutdown_connections 2023-09-30T00:50:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.069+0000 7f56a3b05700 1 --2- 172.21.15.40:0/764939955 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f569c0b1460 0x7f569c0b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:38.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.069+0000 7f56a3b05700 1 --2- 172.21.15.40:0/764939955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f569c0a5fb0 0x7f569c141730 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.069+0000 7f56a3b05700 1 -- 172.21.15.40:0/764939955 >> 172.21.15.40:0/764939955 conn(0x7f569c0631f0 msgr2=0x7f569c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.069+0000 7f56a3b05700 1 -- 172.21.15.40:0/764939955 shutdown_connections 2023-09-30T00:50:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.069+0000 7f56a3b05700 1 -- 172.21.15.40:0/764939955 wait complete. 2023-09-30T00:50:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.070+0000 7f56a3b05700 10 monclient: build_initial_monmap 2023-09-30T00:50:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.070+0000 7f56a3b05700 10 monclient: monmap: 2023-09-30T00:50:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:38.075 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:38.076 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.070+0000 7f56a3b05700 1 librados: starting msgr at 2023-09-30T00:50:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.070+0000 7f56a3b05700 1 librados: starting objecter 2023-09-30T00:50:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.070+0000 7f56a3b05700 1 Processor -- start 2023-09-30T00:50:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.070+0000 7f56a3b05700 1 -- start start 2023-09-30T00:50:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.070+0000 7f56a3b05700 1 librados: setting wanted keys 2023-09-30T00:50:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.070+0000 7f56a3b05700 1 librados: calling monclient init 2023-09-30T00:50:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.070+0000 7f56a3b05700 10 monclient: init 2023-09-30T00:50:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:50:38.076 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f569c0a5fb0 0x7f569c0a9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 10 monclient(hunting): picked mon.noname-c con 0x7f569c0a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f569c0af410 0x7f569c0a88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 10 monclient(hunting): picked mon.noname-a con 0x7f569c0af410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f569c0b1460 0x7f569c151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 10 monclient(hunting): picked mon.noname-b con 0x7f569c0b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c0aade0 con 0x7f569c0af410 2023-09-30T00:50:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c13d880 con 0x7f569c0b1460 2023-09-30T00:50:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569c0aea00 con 0x7f569c0a5fb0 2023-09-30T00:50:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:38.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.071+0000 7f56a3b05700 10 monclient(hunting): authenticate will time out at 5499.372444s 2023-09-30T00:50:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.072+0000 7f56a10a0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f569c0af410 0x7f569c0a88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.072+0000 7f56a10a0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f569c0af410 0x7f569c0a88f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42780/0 (socket says 172.21.15.40:42780) 2023-09-30T00:50:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.072+0000 7f56a10a0700 1 -- 172.21.15.40:0/3439358264 learned_addr learned my addr 172.21.15.40:0/3439358264 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.072+0000 7f56a10a0700 10 monclient(hunting): get_auth_request con 0x7f569c0af410 auth_method 0 2023-09-30T00:50:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.072+0000 7f56a10a0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.072+0000 7f56a10a0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.072+0000 7f56a10a0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.073+0000 7f56a10a0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.073+0000 7f56a10a0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:38.078 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.073+0000 7f56a10a0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:38.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.076+0000 7f56a10a0700 10 monclient(hunting): handle_auth_done global_id 4572 payload 824 2023-09-30T00:50:38.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.077+0000 7f56a10a0700 1 -- 172.21.15.40:0/3439358264 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f569c0a5fb0 msgr2=0x7f569c0a9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:38.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.077+0000 7f56a10a0700 1 --2- 172.21.15.40:0/3439358264 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f569c0a5fb0 0x7f569c0a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:38.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.077+0000 7f56a10a0700 1 -- 172.21.15.40:0/3439358264 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f569c0b1460 msgr2=0x7f569c151640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:38.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.077+0000 7f56a10a0700 1 --2- 172.21.15.40:0/3439358264 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f569c0b1460 0x7f569c151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:38.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.077+0000 7f56a10a0700 10 monclient: _finish_hunting 0 2023-09-30T00:50:38.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.077+0000 7f56a10a0700 1 monclient: found mon.noname-a 2023-09-30T00:50:38.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.077+0000 7f56a10a0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:38.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.077+0000 7f56a10a0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:50:38.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.077+0000 7f56a10a0700 1 -- 172.21.15.40:0/3439358264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c0094f0 con 0x7f569c0af410 2023-09-30T00:50:38.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.079+0000 7f56a10a0700 1 --2- 172.21.15.40:0/3439358264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f569c0af410 0x7f569c0a88f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f568c009290 tx=0x7f568c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:38.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.079+0000 7f5691ffb700 10 client.?.objecter ms_handle_connect 0x7f569c0af410 2023-09-30T00:50:38.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.079+0000 7f5691ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:38.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.081+0000 7f5691ffb700 1 -- 172.21.15.40:0/3439358264 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c026480 con 0x7f569c0af410 2023-09-30T00:50:38.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.081+0000 7f5691ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:38.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.081+0000 7f5691ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:50:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.081+0000 7f5691ffb700 10 monclient: dump: 2023-09-30T00:50:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:38.083 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.081+0000 7f5691ffb700 10 monclient: _finish_auth 0 2023-09-30T00:50:38.083 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.081+0000 7f5691ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.081+0000 7f56a3b05700 5 monclient: authenticate success, global_id 4572 2023-09-30T00:50:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.081+0000 7f56a3b05700 10 monclient: _renew_subs 2023-09-30T00:50:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.081+0000 7f56a3b05700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:50:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.081+0000 7f56a3b05700 1 -- 172.21.15.40:0/3439358264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569c0a7a60 con 0x7f569c0af410 2023-09-30T00:50:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.082+0000 7f56a3b05700 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.082+0000 7f56a3b05700 10 monclient: _renew_subs 2023-09-30T00:50:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.082+0000 7f56a3b05700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:50:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.082+0000 7f56a3b05700 1 -- 172.21.15.40:0/3439358264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569c151dc0 con 0x7f569c0af410 2023-09-30T00:50:38.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.082+0000 7f56a3b05700 1 librados: init done 2023-09-30T00:50:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.083+0000 7f56a3b05700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.083+0000 7f56a3b05700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.083+0000 7f56a3b05700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:50:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.083+0000 7f56a3b05700 1 -- 172.21.15.40:0/3439358264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569c064a00 con 0x7f569c0af410 2023-09-30T00:50:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.084+0000 7f5691ffb700 1 -- 172.21.15.40:0/3439358264 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568c01f070 con 0x7f569c0af410 2023-09-30T00:50:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.084+0000 7f5691ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.084+0000 7f5691ffb700 1 -- 172.21.15.40:0/3439358264 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c0267f0 con 0x7f569c0af410 2023-09-30T00:50:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.084+0000 7f5691ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.084+0000 7f5691ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:50:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.084+0000 7f5693fff700 4 set_mon_vals no callback set 2023-09-30T00:50:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.084+0000 7f5691ffb700 10 monclient: dump: 2023-09-30T00:50:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:38.086 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:38.087 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:38.087 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:38.087 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:38.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.107+0000 7f5691ffb700 1 -- 172.21.15.40:0/3439358264 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f568c01a440 con 0x7f569c0af410 2023-09-30T00:50:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.107+0000 7f5691ffb700 1 --2- 172.21.15.40:0/3439358264 >> v2:172.21.15.135:6800/116426 conn(0x7f56800688c0 0x7f568006ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.108+0000 7f56a18a1700 1 --2- 172.21.15.40:0/3439358264 >> v2:172.21.15.135:6800/116426 conn(0x7f56800688c0 0x7f568006ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.108+0000 7f56a18a1700 10 monclient: get_auth_request con 0x7f56800688c0 auth_method 0 2023-09-30T00:50:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.108+0000 7f5691ffb700 10 client.4572.objecter ms_handle_connect 0x7f56800688c0 2023-09-30T00:50:38.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.108+0000 7f56a18a1700 1 --2- 172.21.15.40:0/3439358264 >> v2:172.21.15.135:6800/116426 conn(0x7f56800688c0 0x7f568006ad70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f56940097e0 tx=0x7f5694006d10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.253+0000 7f5691ffb700 1 -- 172.21.15.40:0/3439358264 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f568c021070 con 0x7f569c0af410 2023-09-30T00:50:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.253+0000 7f5691ffb700 10 client.4572.objecter ms_dispatch 0x7f569c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.253+0000 7f5691ffb700 3 client.4572.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.253+0000 7f5691ffb700 3 client.4572.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.254+0000 7f5691ffb700 20 client.4572.objecter dump_active .. 0 homeless 2023-09-30T00:50:38.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.420+0000 7f5691ffb700 1 -- 172.21.15.40:0/3439358264 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f568c053910 con 0x7f569c0af410 2023-09-30T00:50:38.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.420+0000 7f5691ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:38.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.421+0000 7f5691ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:38.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.679+0000 7f56a3b05700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:38.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.679+0000 7f56a3b05700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:38.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.679+0000 7f56a3b05700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:50:38.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.679+0000 7f56a3b05700 1 -- 172.21.15.40:0/3439358264 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f569c13f740 con 0x7f569c0af410 2023-09-30T00:50:38.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.687+0000 7f5691ffb700 1 -- 172.21.15.40:0/3439358264 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f568c056f30 con 0x7f569c0af410 2023-09-30T00:50:38.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.687+0000 7f5691ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:38.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.687+0000 7f5691ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:38.688 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:38.688 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:38.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 10 librados: watch_flush enter 2023-09-30T00:50:38.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 10 librados: watch_flush exit 2023-09-30T00:50:38.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 20 client.4572.objecter shutdown clearing up homeless session... 2023-09-30T00:50:38.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 10 client.4572.objecter successfully canceled tick 2023-09-30T00:50:38.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 1 -- 172.21.15.40:0/3439358264 >> v2:172.21.15.135:6800/116426 conn(0x7f56800688c0 msgr2=0x7f568006ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:38.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 1 --2- 172.21.15.40:0/3439358264 >> v2:172.21.15.135:6800/116426 conn(0x7f56800688c0 0x7f568006ad70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f56940097e0 tx=0x7f5694006d10 comp rx=0 tx=0).stop 2023-09-30T00:50:38.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 10 monclient: shutdown 2023-09-30T00:50:38.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:38.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 1 -- 172.21.15.40:0/3439358264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f569c0af410 msgr2=0x7f569c0a88f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:38.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 1 --2- 172.21.15.40:0/3439358264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f569c0af410 0x7f569c0a88f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f568c009290 tx=0x7f568c016000 comp rx=0 tx=0).stop 2023-09-30T00:50:38.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 1 -- 172.21.15.40:0/3439358264 shutdown_connections 2023-09-30T00:50:38.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 1 --2- 172.21.15.40:0/3439358264 >> v2:172.21.15.135:6800/116426 conn(0x7f56800688c0 0x7f568006ad70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:38.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 1 --2- 172.21.15.40:0/3439358264 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f569c0b1460 0x7f569c151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:38.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 1 --2- 172.21.15.40:0/3439358264 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f569c0a5fb0 0x7f569c0a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:38.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 1 --2- 172.21.15.40:0/3439358264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f569c0af410 0x7f569c0a88f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:38.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 1 -- 172.21.15.40:0/3439358264 >> 172.21.15.40:0/3439358264 conn(0x7f569c0631f0 msgr2=0x7f569c13b580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:38.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 1 -- 172.21.15.40:0/3439358264 shutdown_connections 2023-09-30T00:50:38.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.689+0000 7f56a3b05700 1 -- 172.21.15.40:0/3439358264 wait complete. 2023-09-30T00:50:38.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:38.690+0000 7f56a3b05700 1 librados: shutdown 2023-09-30T00:50:41.704 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:41.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.839+0000 7f5569b86700 10 monclient: get_monmap_and_config 2023-09-30T00:50:41.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.839+0000 7f5569b86700 10 monclient: build_initial_monmap 2023-09-30T00:50:41.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.839+0000 7f5569b86700 10 monclient: monmap: 2023-09-30T00:50:41.840 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:41.840 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:41.840 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:41.840477+0000 2023-09-30T00:50:41.840 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:41.840477+0000 2023-09-30T00:50:41.840 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:41.840 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:41.841 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:41.841 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.840+0000 7f5569b86700 1 Processor -- start 2023-09-30T00:50:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.840+0000 7f5569b86700 1 -- start start 2023-09-30T00:50:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.840+0000 7f5569b86700 10 monclient: init 2023-09-30T00:50:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5569b86700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5569b86700 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-09-30T00:50:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5569b86700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55640b1460 0x7f55640b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5569b86700 10 monclient(hunting): picked mon.noname-b con 0x7f55640b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5569b86700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55640a5fb0 0x7f5564141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5569b86700 10 monclient(hunting): picked mon.noname-a con 0x7f55640a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:50:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5569b86700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f55640af410 addr v1:172.21.15.135:6789/0 2023-09-30T00:50:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5569b86700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55640a71f0 con 0x7f55640af410 2023-09-30T00:50:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5569b86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5569b86700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55640a74d0 con 0x7f55640a5fb0 2023-09-30T00:50:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5569b86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5569b86700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55640a63c0 con 0x7f55640b1460 2023-09-30T00:50:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5569b86700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5569b86700 10 monclient(hunting): authenticate will time out at 5503.142184s 2023-09-30T00:50:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f55637fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55640b1460 0x7f55640b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.841+0000 7f5562ffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55640a5fb0 0x7f5564141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.842+0000 7f5563fff700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f55640af410 0x7f55640af890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:48680/0 (socket says 172.21.15.40:48680) 2023-09-30T00:50:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.842+0000 7f5563fff700 1 -- 172.21.15.40:0/2125222788 learned_addr learned my addr 172.21.15.40:0/2125222788 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:50:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.842+0000 7f55637fe700 10 monclient(hunting): get_auth_request con 0x7f55640b1460 auth_method 0 2023-09-30T00:50:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.842+0000 7f55637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.842+0000 7f55637fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.842+0000 7f55637fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.842+0000 7f5562ffd700 10 monclient(hunting): get_auth_request con 0x7f55640a5fb0 auth_method 0 2023-09-30T00:50:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.842+0000 7f5562ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:41.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.842+0000 7f5562ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.842+0000 7f5562ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.843+0000 7f55637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.843+0000 7f55637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.843+0000 7f55637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.843+0000 7f5562ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.843+0000 7f5562ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:41.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.843+0000 7f5562ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.845+0000 7f55637fe700 10 monclient(hunting): handle_auth_done global_id 4511 payload 274 2023-09-30T00:50:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.845+0000 7f55637fe700 1 -- 172.21.15.40:0/2125222788 >> v2:172.21.15.40:3300/0 conn(0x7f55640a5fb0 msgr2=0x7f5564141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.845+0000 7f55637fe700 1 --2- 172.21.15.40:0/2125222788 >> v2:172.21.15.40:3300/0 conn(0x7f55640a5fb0 0x7f5564141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.845+0000 7f55637fe700 1 -- 172.21.15.40:0/2125222788 >> v1:172.21.15.135:6789/0 conn(0x7f55640af410 legacy=0x7f55640af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.845+0000 7f55637fe700 10 monclient: _finish_hunting 0 2023-09-30T00:50:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.845+0000 7f55637fe700 1 monclient: found mon.noname-b 2023-09-30T00:50:41.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.845+0000 7f55637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.845+0000 7f55637fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:50:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.845+0000 7f55637fe700 1 -- 172.21.15.40:0/2125222788 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55640a56d0 con 0x7f55640b1460 2023-09-30T00:50:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.848+0000 7f55637fe700 1 --2- 172.21.15.40:0/2125222788 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55640b1460 0x7f55640b1830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f55540098e0 tx=0x7f55540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7e2c82533fcdbbf server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:41.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f55627fc700 1 -- 172.21.15.40:0/2125222788 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555400b640 con 0x7f55640b1460 2023-09-30T00:50:41.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f55627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:41.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f55627fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:41.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f55627fc700 10 monclient: dump: 2023-09-30T00:50:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:41.854 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:41.855 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f55627fc700 10 monclient: _finish_auth 0 2023-09-30T00:50:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f55627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f55627fc700 1 -- 172.21.15.40:0/2125222788 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555400b7a0 con 0x7f55640b1460 2023-09-30T00:50:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f55627fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f55627fc700 1 -- 172.21.15.40:0/2125222788 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555400bb40 con 0x7f55640b1460 2023-09-30T00:50:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f5569b86700 5 monclient: authenticate success, global_id 4511 2023-09-30T00:50:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f5569b86700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f5569b86700 4 set_mon_vals no callback set 2023-09-30T00:50:41.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f55627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f55627fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:50:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.852+0000 7f55627fc700 10 monclient: dump: 2023-09-30T00:50:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:41.856 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:41.857 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:41.857 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:41.857 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:41.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.853+0000 7f5569b86700 10 monclient: shutdown 2023-09-30T00:50:41.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.853+0000 7f5569b86700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:41.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.853+0000 7f5569b86700 1 -- 172.21.15.40:0/2125222788 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55640b1460 msgr2=0x7f55640b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:41.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.853+0000 7f5569b86700 1 --2- 172.21.15.40:0/2125222788 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55640b1460 0x7f55640b1830 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f55540098e0 tx=0x7f55540091a0 comp rx=0 tx=0).stop 2023-09-30T00:50:41.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.853+0000 7f5569b86700 1 -- 172.21.15.40:0/2125222788 shutdown_connections 2023-09-30T00:50:41.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.853+0000 7f5569b86700 1 --2- 172.21.15.40:0/2125222788 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55640b1460 0x7f55640b1830 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:41.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.853+0000 7f5569b86700 1 --2- 172.21.15.40:0/2125222788 >> v2:172.21.15.40:3300/0 conn(0x7f55640a5fb0 0x7f5564141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:41.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.853+0000 7f5569b86700 1 -- 172.21.15.40:0/2125222788 >> 172.21.15.40:0/2125222788 conn(0x7f55640631f0 msgr2=0x7f55640635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:41.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.853+0000 7f5569b86700 1 -- 172.21.15.40:0/2125222788 shutdown_connections 2023-09-30T00:50:41.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.853+0000 7f5569b86700 1 -- 172.21.15.40:0/2125222788 wait complete. 2023-09-30T00:50:41.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.854+0000 7f5569b86700 10 monclient: build_initial_monmap 2023-09-30T00:50:41.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.854+0000 7f5569b86700 10 monclient: monmap: 2023-09-30T00:50:41.862 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:41.862 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:41.862 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:41.862 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:41.862 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:41.862 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:41.862 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:41.863 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:41.863 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:41.863 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:41.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.854+0000 7f5569b86700 1 librados: starting msgr at 2023-09-30T00:50:41.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.854+0000 7f5569b86700 1 librados: starting objecter 2023-09-30T00:50:41.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.854+0000 7f5569b86700 1 Processor -- start 2023-09-30T00:50:41.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 1 -- start start 2023-09-30T00:50:41.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 1 librados: setting wanted keys 2023-09-30T00:50:41.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 1 librados: calling monclient init 2023-09-30T00:50:41.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 10 monclient: init 2023-09-30T00:50:41.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:41.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:50:41.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55640a5fb0 0x7f55640a8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:41.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 10 monclient(hunting): picked mon.noname-b con 0x7f55640a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:41.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55640af410 0x7f55640a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:41.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 10 monclient(hunting): picked mon.noname-a con 0x7f55640af410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:41.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55640b1460 0x7f556414db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:41.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 10 monclient(hunting): picked mon.noname-c con 0x7f55640b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:41.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:41.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55640aa5f0 con 0x7f55640af410 2023-09-30T00:50:41.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:41.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55640a9540 con 0x7f55640a5fb0 2023-09-30T00:50:41.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:41.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556413eb20 con 0x7f55640b1460 2023-09-30T00:50:41.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:41.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.855+0000 7f5569b86700 10 monclient(hunting): authenticate will time out at 5503.156657s 2023-09-30T00:50:41.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f5562ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55640af410 0x7f55640a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:41.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f55637fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55640a5fb0 0x7f55640a8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:41.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f5563fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55640b1460 0x7f556414db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:41.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f5562ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55640af410 0x7f55640a8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40368/0 (socket says 172.21.15.40:40368) 2023-09-30T00:50:41.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f5562ffd700 1 -- 172.21.15.40:0/4191364423 learned_addr learned my addr 172.21.15.40:0/4191364423 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:41.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f5562ffd700 10 monclient(hunting): get_auth_request con 0x7f55640af410 auth_method 0 2023-09-30T00:50:41.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f5562ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:41.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f5562ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:41.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f5562ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:41.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f5563fff700 10 monclient(hunting): get_auth_request con 0x7f55640b1460 auth_method 0 2023-09-30T00:50:41.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f5563fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:41.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f5563fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:41.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f5563fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:41.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f55637fe700 10 monclient(hunting): get_auth_request con 0x7f55640a5fb0 auth_method 0 2023-09-30T00:50:41.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f55637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:41.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f55637fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:41.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.856+0000 7f55637fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:41.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.857+0000 7f5562ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:41.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.857+0000 7f5562ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:41.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.857+0000 7f5562ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:41.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.857+0000 7f5563fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:41.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.857+0000 7f5563fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:41.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.857+0000 7f5563fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:41.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.857+0000 7f55637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:41.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.857+0000 7f55637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:41.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.857+0000 7f55637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:41.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.861+0000 7f5562ffd700 10 monclient(hunting): handle_auth_done global_id 4578 payload 824 2023-09-30T00:50:41.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.861+0000 7f5562ffd700 1 -- 172.21.15.40:0/4191364423 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55640b1460 msgr2=0x7f556414db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:41.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.861+0000 7f5562ffd700 1 --2- 172.21.15.40:0/4191364423 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55640b1460 0x7f556414db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:41.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.861+0000 7f5562ffd700 1 -- 172.21.15.40:0/4191364423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55640a5fb0 msgr2=0x7f55640a8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:41.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.861+0000 7f5562ffd700 1 --2- 172.21.15.40:0/4191364423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55640a5fb0 0x7f55640a8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:41.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.861+0000 7f5562ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:50:41.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.861+0000 7f5562ffd700 1 monclient: found mon.noname-a 2023-09-30T00:50:41.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.861+0000 7f5562ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:41.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.861+0000 7f5562ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:50:41.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.861+0000 7f5562ffd700 1 -- 172.21.15.40:0/4191364423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5554009420 con 0x7f55640af410 2023-09-30T00:50:41.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.863+0000 7f5562ffd700 1 --2- 172.21.15.40:0/4191364423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55640af410 0x7f55640a8e40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f555800ec80 tx=0x7f55580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:41.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.864+0000 7f554b7fe700 10 client.?.objecter ms_handle_connect 0x7f55640af410 2023-09-30T00:50:41.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.864+0000 7f554b7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:41.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f554b7fe700 1 -- 172.21.15.40:0/4191364423 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555800c850 con 0x7f55640af410 2023-09-30T00:50:41.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f554b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:41.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f554b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:50:41.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f554b7fe700 10 monclient: dump: 2023-09-30T00:50:41.872 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:41.872 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:41.872 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:41.872 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:41.872 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:41.872 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:41.872 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:41.872 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:41.873 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:41.873 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:41.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f554b7fe700 10 monclient: _finish_auth 0 2023-09-30T00:50:41.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f554b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:41.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f554b7fe700 1 -- 172.21.15.40:0/4191364423 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555800c9b0 con 0x7f55640af410 2023-09-30T00:50:41.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f5569b86700 5 monclient: authenticate success, global_id 4578 2023-09-30T00:50:41.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f5569b86700 10 monclient: _renew_subs 2023-09-30T00:50:41.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f5569b86700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:50:41.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f5569b86700 1 -- 172.21.15.40:0/4191364423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556414e0c0 con 0x7f55640af410 2023-09-30T00:50:41.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f554b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:41.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f554b7fe700 1 -- 172.21.15.40:0/4191364423 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555800cd50 con 0x7f55640af410 2023-09-30T00:50:41.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f554b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:41.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f554b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:50:41.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f554b7fe700 10 monclient: dump: 2023-09-30T00:50:41.874 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:41.874 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:41.874 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:41.874 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:41.874 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:41.874 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:41.874 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:41.875 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:41.875 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:41.875 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:41.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f5561ffb700 4 set_mon_vals no callback set 2023-09-30T00:50:41.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f5569b86700 10 client.4578.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:41.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f5569b86700 10 monclient: _renew_subs 2023-09-30T00:50:41.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f5569b86700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:50:41.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f5569b86700 1 -- 172.21.15.40:0/4191364423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556414e600 con 0x7f55640af410 2023-09-30T00:50:41.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.868+0000 7f5569b86700 1 librados: init done 2023-09-30T00:50:41.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.869+0000 7f5569b86700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:41.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.869+0000 7f5569b86700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:41.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.869+0000 7f5569b86700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:50:41.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:41.869+0000 7f5569b86700 1 -- 172.21.15.40:0/4191364423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55640aa460 con 0x7f55640af410 2023-09-30T00:50:42.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.168+0000 7f554b7fe700 1 -- 172.21.15.40:0/4191364423 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5558013680 con 0x7f55640af410 2023-09-30T00:50:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.168+0000 7f554b7fe700 1 --2- 172.21.15.40:0/4191364423 >> v2:172.21.15.135:6800/116426 conn(0x7f55500688c0 0x7f555006ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.168+0000 7f554b7fe700 1 -- 172.21.15.40:0/4191364423 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f55580848b0 con 0x7f55640af410 2023-09-30T00:50:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.168+0000 7f554b7fe700 10 client.4578.objecter ms_dispatch 0x7f5564001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.168+0000 7f554b7fe700 3 client.4578.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.168+0000 7f554b7fe700 3 client.4578.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.169+0000 7f554b7fe700 20 client.4578.objecter dump_active .. 0 homeless 2023-09-30T00:50:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.169+0000 7f554b7fe700 1 -- 172.21.15.40:0/4191364423 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f55580af640 con 0x7f55640af410 2023-09-30T00:50:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.169+0000 7f554b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.169+0000 7f554b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:42.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.169+0000 7f55637fe700 1 --2- 172.21.15.40:0/4191364423 >> v2:172.21.15.135:6800/116426 conn(0x7f55500688c0 0x7f555006ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:42.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.169+0000 7f55637fe700 10 monclient: get_auth_request con 0x7f55500688c0 auth_method 0 2023-09-30T00:50:42.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.169+0000 7f55637fe700 1 --2- 172.21.15.40:0/4191364423 >> v2:172.21.15.135:6800/116426 conn(0x7f55500688c0 0x7f555006ad70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f55540093f0 tx=0x7f5554017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:42.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.169+0000 7f554b7fe700 10 client.4578.objecter ms_handle_connect 0x7f55500688c0 2023-09-30T00:50:42.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.439+0000 7f5569b86700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:42.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.439+0000 7f5569b86700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:42.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.439+0000 7f5569b86700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:50:42.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.439+0000 7f5569b86700 1 -- 172.21.15.40:0/4191364423 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f556414d8a0 con 0x7f55640af410 2023-09-30T00:50:42.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.458+0000 7f554b7fe700 1 -- 172.21.15.40:0/4191364423 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f556414d8a0 con 0x7f55640af410 2023-09-30T00:50:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.458+0000 7f554b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:42.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.458+0000 7f554b7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:42.459 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:42.459 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:42.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.460+0000 7f5569b86700 10 librados: watch_flush enter 2023-09-30T00:50:42.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.460+0000 7f5569b86700 10 librados: watch_flush exit 2023-09-30T00:50:42.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.460+0000 7f5569b86700 20 client.4578.objecter shutdown clearing up homeless session... 2023-09-30T00:50:42.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.460+0000 7f5569b86700 10 client.4578.objecter successfully canceled tick 2023-09-30T00:50:42.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.460+0000 7f5569b86700 1 -- 172.21.15.40:0/4191364423 >> v2:172.21.15.135:6800/116426 conn(0x7f55500688c0 msgr2=0x7f555006ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:42.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.460+0000 7f5569b86700 1 --2- 172.21.15.40:0/4191364423 >> v2:172.21.15.135:6800/116426 conn(0x7f55500688c0 0x7f555006ad70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f55540093f0 tx=0x7f5554017000 comp rx=0 tx=0).stop 2023-09-30T00:50:42.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.460+0000 7f5569b86700 10 monclient: shutdown 2023-09-30T00:50:42.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.460+0000 7f5569b86700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:42.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.460+0000 7f5569b86700 1 -- 172.21.15.40:0/4191364423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55640af410 msgr2=0x7f55640a8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:42.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.460+0000 7f5569b86700 1 --2- 172.21.15.40:0/4191364423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55640af410 0x7f55640a8e40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f555800ec80 tx=0x7f55580091a0 comp rx=0 tx=0).stop 2023-09-30T00:50:42.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.461+0000 7f5569b86700 1 -- 172.21.15.40:0/4191364423 shutdown_connections 2023-09-30T00:50:42.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.461+0000 7f5569b86700 1 --2- 172.21.15.40:0/4191364423 >> v2:172.21.15.135:6800/116426 conn(0x7f55500688c0 0x7f555006ad70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:42.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.461+0000 7f5569b86700 1 --2- 172.21.15.40:0/4191364423 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f55640a5fb0 0x7f55640a8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:42.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.461+0000 7f5569b86700 1 --2- 172.21.15.40:0/4191364423 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f55640b1460 0x7f556414db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:42.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.461+0000 7f5569b86700 1 --2- 172.21.15.40:0/4191364423 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55640af410 0x7f55640a8e40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:42.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.461+0000 7f5569b86700 1 -- 172.21.15.40:0/4191364423 >> 172.21.15.40:0/4191364423 conn(0x7f55640631f0 msgr2=0x7f5564144b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:42.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.461+0000 7f5569b86700 1 -- 172.21.15.40:0/4191364423 shutdown_connections 2023-09-30T00:50:42.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.461+0000 7f5569b86700 1 -- 172.21.15.40:0/4191364423 wait complete. 2023-09-30T00:50:42.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:42.461+0000 7f5569b86700 1 librados: shutdown 2023-09-30T00:50:45.477 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:45.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.617+0000 7f8de0129700 10 monclient: get_monmap_and_config 2023-09-30T00:50:45.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.617+0000 7f8de0129700 10 monclient: build_initial_monmap 2023-09-30T00:50:45.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.617+0000 7f8de0129700 10 monclient: monmap: 2023-09-30T00:50:45.618 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:45.618 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:45.618 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:45.618550+0000 2023-09-30T00:50:45.618 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:45.618550+0000 2023-09-30T00:50:45.618 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:45.618 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:45.619 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:45.619 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:45.619 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:45.619 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:45.619 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:45.619 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:45.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.618+0000 7f8de0129700 1 Processor -- start 2023-09-30T00:50:45.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.618+0000 7f8de0129700 1 -- start start 2023-09-30T00:50:45.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.618+0000 7f8de0129700 10 monclient: init 2023-09-30T00:50:45.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.618+0000 7f8de0129700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:45.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.618+0000 7f8de0129700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-09-30T00:50:45.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.618+0000 7f8de0129700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8dd80b1460 0x7f8dd80b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:45.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.618+0000 7f8de0129700 10 monclient(hunting): picked mon.noname-a con 0x7f8dd80b1460 addr v2:172.21.15.40:3300/0 2023-09-30T00:50:45.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.618+0000 7f8de0129700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8dd80a5fb0 addr v1:172.21.15.40:6789/0 2023-09-30T00:50:45.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.618+0000 7f8de0129700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 0x7f8dd813b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:45.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.618+0000 7f8de0129700 10 monclient(hunting): picked mon.noname-b con 0x7f8dd813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:45.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.618+0000 7f8de0129700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dd80a5b20 con 0x7f8dd80a5fb0 2023-09-30T00:50:45.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.618+0000 7f8de0129700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:45.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.618+0000 7f8de0129700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd80a71f0 con 0x7f8dd80b1460 2023-09-30T00:50:45.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.619+0000 7f8de0129700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:45.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.619+0000 7f8de0129700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80a74d0 con 0x7f8dd813ac20 2023-09-30T00:50:45.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.619+0000 7f8de0129700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:45.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.619+0000 7f8de0129700 10 monclient(hunting): authenticate will time out at 5506.920018s 2023-09-30T00:50:45.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.619+0000 7f8dde6c6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 0x7f8dd813b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:45.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.619+0000 7f8dde6c6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 0x7f8dd813b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:35730/0 (socket says 172.21.15.40:35730) 2023-09-30T00:50:45.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.620+0000 7f8dde6c6700 1 -- 172.21.15.40:0/3406181088 learned_addr learned my addr 172.21.15.40:0/3406181088 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:45.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.620+0000 7f8dde6c6700 10 monclient(hunting): get_auth_request con 0x7f8dd813ac20 auth_method 0 2023-09-30T00:50:45.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.620+0000 7f8dde6c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:45.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.620+0000 7f8dde6c6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:45.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.620+0000 7f8dde6c6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:45.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.620+0000 7f8dddec5700 1 --2- 172.21.15.40:0/3406181088 >> v2:172.21.15.40:3300/0 conn(0x7f8dd80b1460 0x7f8dd80b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:45.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.620+0000 7f8dde6c6700 1 -- 172.21.15.40:0/3406181088 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 msgr2=0x7f8dd813b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T00:50:45.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.620+0000 7f8dde6c6700 1 -- 172.21.15.40:0/3406181088 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 msgr2=0x7f8dd813b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:50:45.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.620+0000 7f8dde6c6700 1 --2- 172.21.15.40:0/3406181088 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 0x7f8dd813b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:50:45.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.620+0000 7f8dde6c6700 1 --2- 172.21.15.40:0/3406181088 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 0x7f8dd813b080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:50:45.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.620+0000 7f8dddec5700 10 monclient(hunting): get_auth_request con 0x7f8dd80b1460 auth_method 0 2023-09-30T00:50:45.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.620+0000 7f8dddec5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:45.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.620+0000 7f8dddec5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:45.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.620+0000 7f8dddec5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:45.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.621+0000 7f8dddec5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:45.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.621+0000 7f8dddec5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:45.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.621+0000 7f8dddec5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:45.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.624+0000 7f8dddec5700 10 monclient(hunting): handle_auth_done global_id 4581 payload 274 2023-09-30T00:50:45.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.624+0000 7f8dddec5700 1 -- 172.21.15.40:0/3406181088 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 msgr2=0x7f8dd813b080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:50:45.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.624+0000 7f8dddec5700 1 --2- 172.21.15.40:0/3406181088 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 0x7f8dd813b080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:45.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.624+0000 7f8dddec5700 1 -- 172.21.15.40:0/3406181088 >> v1:172.21.15.40:6789/0 conn(0x7f8dd80a5fb0 legacy=0x7f8dd8141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:45.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.624+0000 7f8dddec5700 10 monclient: _finish_hunting 0 2023-09-30T00:50:45.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.624+0000 7f8dddec5700 1 monclient: found mon.noname-a 2023-09-30T00:50:45.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.624+0000 7f8dddec5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:45.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.624+0000 7f8dddec5700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:50:45.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.624+0000 7f8dddec5700 1 -- 172.21.15.40:0/3406181088 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd80a63c0 con 0x7f8dd80b1460 2023-09-30T00:50:45.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.626+0000 7f8dddec5700 1 --2- 172.21.15.40:0/3406181088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd80b1460 0x7f8dd80b1830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40098e0 tx=0x7f8dd40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f387a4e44e9a637d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8ddcec3700 1 -- 172.21.15.40:0/3406181088 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4009e80 con 0x7f8dd80b1460 2023-09-30T00:50:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8ddcec3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8ddcec3700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:50:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8ddcec3700 10 monclient: dump: 2023-09-30T00:50:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:45.633 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8ddcec3700 10 monclient: _finish_auth 0 2023-09-30T00:50:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8ddcec3700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8ddcec3700 1 -- 172.21.15.40:0/3406181088 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd400b820 con 0x7f8dd80b1460 2023-09-30T00:50:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8ddcec3700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8ddcec3700 1 -- 172.21.15.40:0/3406181088 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd400bba0 con 0x7f8dd80b1460 2023-09-30T00:50:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8ddcec3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8ddcec3700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:50:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8ddcec3700 10 monclient: dump: 2023-09-30T00:50:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:45.634 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:45.635 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8de0129700 5 monclient: authenticate success, global_id 4581 2023-09-30T00:50:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8de0129700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.632+0000 7f8de0129700 4 set_mon_vals no callback set 2023-09-30T00:50:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.633+0000 7f8de0129700 10 monclient: shutdown 2023-09-30T00:50:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.633+0000 7f8de0129700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.633+0000 7f8de0129700 1 -- 172.21.15.40:0/3406181088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd80b1460 msgr2=0x7f8dd80b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.633+0000 7f8de0129700 1 --2- 172.21.15.40:0/3406181088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd80b1460 0x7f8dd80b1830 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40098e0 tx=0x7f8dd40091a0 comp rx=0 tx=0).stop 2023-09-30T00:50:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.633+0000 7f8de0129700 1 -- 172.21.15.40:0/3406181088 shutdown_connections 2023-09-30T00:50:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.633+0000 7f8de0129700 1 --2- 172.21.15.40:0/3406181088 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 0x7f8dd813b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.633+0000 7f8de0129700 1 --2- 172.21.15.40:0/3406181088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd80b1460 0x7f8dd80b1830 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.633+0000 7f8de0129700 1 -- 172.21.15.40:0/3406181088 >> 172.21.15.40:0/3406181088 conn(0x7f8dd80631f0 msgr2=0x7f8dd80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.633+0000 7f8de0129700 1 -- 172.21.15.40:0/3406181088 shutdown_connections 2023-09-30T00:50:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.633+0000 7f8de0129700 1 -- 172.21.15.40:0/3406181088 wait complete. 2023-09-30T00:50:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.634+0000 7f8de0129700 10 monclient: build_initial_monmap 2023-09-30T00:50:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.634+0000 7f8de0129700 10 monclient: monmap: 2023-09-30T00:50:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:45.640 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.634+0000 7f8de0129700 1 librados: starting msgr at 2023-09-30T00:50:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.634+0000 7f8de0129700 1 librados: starting objecter 2023-09-30T00:50:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.634+0000 7f8de0129700 1 Processor -- start 2023-09-30T00:50:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 1 -- start start 2023-09-30T00:50:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 1 librados: setting wanted keys 2023-09-30T00:50:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 1 librados: calling monclient init 2023-09-30T00:50:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 10 monclient: init 2023-09-30T00:50:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:50:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd80a5fb0 0x7f8dd813e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 10 monclient(hunting): picked mon.noname-a con 0x7f8dd80a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8dd80b1460 0x7f8dd80a79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 10 monclient(hunting): picked mon.noname-b con 0x7f8dd80b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 0x7f8dd80a7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 10 monclient(hunting): picked mon.noname-c con 0x7f8dd813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80a8430 con 0x7f8dd80a5fb0 2023-09-30T00:50:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80aa960 con 0x7f8dd80b1460 2023-09-30T00:50:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd80abae0 con 0x7f8dd813ac20 2023-09-30T00:50:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.635+0000 7f8de0129700 10 monclient(hunting): authenticate will time out at 5506.936808s 2023-09-30T00:50:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.636+0000 7f8dde6c6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 0x7f8dd80a7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.636+0000 7f8dddec5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd80a5fb0 0x7f8dd813e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.636+0000 7f8ddd6c4700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8dd80b1460 0x7f8dd80a79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.636+0000 7f8dde6c6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 0x7f8dd80a7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:35744/0 (socket says 172.21.15.40:35744) 2023-09-30T00:50:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.636+0000 7f8dde6c6700 1 -- 172.21.15.40:0/303246784 learned_addr learned my addr 172.21.15.40:0/303246784 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.636+0000 7f8dde6c6700 10 monclient(hunting): get_auth_request con 0x7f8dd813ac20 auth_method 0 2023-09-30T00:50:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.636+0000 7f8dde6c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.636+0000 7f8dde6c6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.636+0000 7f8dde6c6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.637+0000 7f8dddec5700 10 monclient(hunting): get_auth_request con 0x7f8dd80a5fb0 auth_method 0 2023-09-30T00:50:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.637+0000 7f8dddec5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.637+0000 7f8dddec5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.637+0000 7f8dddec5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.637+0000 7f8ddd6c4700 10 monclient(hunting): get_auth_request con 0x7f8dd80b1460 auth_method 0 2023-09-30T00:50:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.637+0000 7f8ddd6c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.637+0000 7f8ddd6c4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.637+0000 7f8ddd6c4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.637+0000 7f8dde6c6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.637+0000 7f8dde6c6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:45.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.637+0000 7f8dde6c6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.637+0000 7f8dddec5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.637+0000 7f8dddec5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.637+0000 7f8dddec5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.638+0000 7f8ddd6c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.638+0000 7f8ddd6c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.638+0000 7f8ddd6c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.641+0000 7f8dddec5700 10 monclient(hunting): handle_auth_done global_id 4584 payload 824 2023-09-30T00:50:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.641+0000 7f8dddec5700 1 -- 172.21.15.40:0/303246784 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 msgr2=0x7f8dd80a7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.641+0000 7f8dddec5700 1 --2- 172.21.15.40:0/303246784 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 0x7f8dd80a7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.641+0000 7f8dddec5700 1 -- 172.21.15.40:0/303246784 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8dd80b1460 msgr2=0x7f8dd80a79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.641+0000 7f8dddec5700 1 --2- 172.21.15.40:0/303246784 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8dd80b1460 0x7f8dd80a79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.641+0000 7f8dddec5700 10 monclient: _finish_hunting 0 2023-09-30T00:50:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.641+0000 7f8dddec5700 1 monclient: found mon.noname-a 2023-09-30T00:50:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.641+0000 7f8dddec5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.641+0000 7f8dddec5700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:50:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.641+0000 7f8dddec5700 1 -- 172.21.15.40:0/303246784 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd4009420 con 0x7f8dd80a5fb0 2023-09-30T00:50:45.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.644+0000 7f8dddec5700 1 --2- 172.21.15.40:0/303246784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd80a5fb0 0x7f8dd813e460 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4015000 tx=0x7f8dd4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:45.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.644+0000 7f8dcdffb700 10 client.?.objecter ms_handle_connect 0x7f8dd80a5fb0 2023-09-30T00:50:45.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.644+0000 7f8dcdffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8dcdffb700 1 -- 172.21.15.40:0/303246784 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd400b990 con 0x7f8dd80a5fb0 2023-09-30T00:50:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8dcdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8dcdffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:50:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8dcdffb700 10 monclient: dump: 2023-09-30T00:50:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:45.650 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:45.650 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:45.650 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:45.650 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:45.650 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:45.650 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:45.650 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:45.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8dcdffb700 10 monclient: _finish_auth 0 2023-09-30T00:50:45.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8dcdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:45.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8dcdffb700 1 -- 172.21.15.40:0/303246784 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd4020070 con 0x7f8dd80a5fb0 2023-09-30T00:50:45.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8dcdffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:45.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8dcdffb700 1 -- 172.21.15.40:0/303246784 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd401b450 con 0x7f8dd80a5fb0 2023-09-30T00:50:45.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8de0129700 5 monclient: authenticate success, global_id 4584 2023-09-30T00:50:45.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8de0129700 10 monclient: _renew_subs 2023-09-30T00:50:45.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8de0129700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:50:45.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8de0129700 1 -- 172.21.15.40:0/303246784 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd80aa440 con 0x7f8dd80a5fb0 2023-09-30T00:50:45.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8dcdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:45.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8dcdffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:50:45.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8dcffff700 4 set_mon_vals no callback set 2023-09-30T00:50:45.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8dcdffb700 10 monclient: dump: 2023-09-30T00:50:45.651 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:45.651 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:45.651 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:45.651 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:45.651 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:45.652 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8de0129700 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8de0129700 10 monclient: _renew_subs 2023-09-30T00:50:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8de0129700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:50:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8de0129700 1 -- 172.21.15.40:0/303246784 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd80aa1d0 con 0x7f8dd80a5fb0 2023-09-30T00:50:45.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.648+0000 7f8de0129700 1 librados: init done 2023-09-30T00:50:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.649+0000 7f8de0129700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.649+0000 7f8de0129700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.649+0000 7f8de0129700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:50:45.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.649+0000 7f8de0129700 1 -- 172.21.15.40:0/303246784 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd8064a00 con 0x7f8dd80a5fb0 2023-09-30T00:50:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.882+0000 7f8dcdffb700 1 -- 172.21.15.40:0/303246784 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8dd4022030 con 0x7f8dd80a5fb0 2023-09-30T00:50:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.883+0000 7f8dcdffb700 1 --2- 172.21.15.40:0/303246784 >> v2:172.21.15.135:6800/116426 conn(0x7f8dbc068960 0x7f8dbc06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.883+0000 7f8ddd6c4700 1 --2- 172.21.15.40:0/303246784 >> v2:172.21.15.135:6800/116426 conn(0x7f8dbc068960 0x7f8dbc06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.883+0000 7f8ddd6c4700 10 monclient: get_auth_request con 0x7f8dbc068960 auth_method 0 2023-09-30T00:50:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.884+0000 7f8dcdffb700 10 client.4584.objecter ms_handle_connect 0x7f8dbc068960 2023-09-30T00:50:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.884+0000 7f8ddd6c4700 1 --2- 172.21.15.40:0/303246784 >> v2:172.21.15.135:6800/116426 conn(0x7f8dbc068960 0x7f8dbc06ae10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8dd80ab450 tx=0x7f8dc80090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.914+0000 7f8dcdffb700 1 -- 172.21.15.40:0/303246784 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8dd4057c80 con 0x7f8dd80a5fb0 2023-09-30T00:50:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.914+0000 7f8dcdffb700 10 client.4584.objecter ms_dispatch 0x7f8dd8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.914+0000 7f8dcdffb700 3 client.4584.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.914+0000 7f8dcdffb700 3 client.4584.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:45.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.915+0000 7f8dcdffb700 20 client.4584.objecter dump_active .. 0 homeless 2023-09-30T00:50:45.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.954+0000 7f8dcdffb700 1 -- 172.21.15.40:0/303246784 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8dd40538c0 con 0x7f8dd80a5fb0 2023-09-30T00:50:45.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.954+0000 7f8dcdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:45.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:45.954+0000 7f8dcdffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:46.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.223+0000 7f8de0129700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:46.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.223+0000 7f8de0129700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:46.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.223+0000 7f8de0129700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:50:46.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.223+0000 7f8de0129700 1 -- 172.21.15.40:0/303246784 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8dd8064a00 con 0x7f8dd80a5fb0 2023-09-30T00:50:46.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.229+0000 7f8dcdffb700 1 -- 172.21.15.40:0/303246784 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8dd4056ee0 con 0x7f8dd80a5fb0 2023-09-30T00:50:46.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.229+0000 7f8dcdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:46.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.229+0000 7f8dcdffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:46.230 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:46.230 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:46.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 10 librados: watch_flush enter 2023-09-30T00:50:46.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 10 librados: watch_flush exit 2023-09-30T00:50:46.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 20 client.4584.objecter shutdown clearing up homeless session... 2023-09-30T00:50:46.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 10 client.4584.objecter successfully canceled tick 2023-09-30T00:50:46.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 1 -- 172.21.15.40:0/303246784 >> v2:172.21.15.135:6800/116426 conn(0x7f8dbc068960 msgr2=0x7f8dbc06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:46.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 1 --2- 172.21.15.40:0/303246784 >> v2:172.21.15.135:6800/116426 conn(0x7f8dbc068960 0x7f8dbc06ae10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8dd80ab450 tx=0x7f8dc80090f0 comp rx=0 tx=0).stop 2023-09-30T00:50:46.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 10 monclient: shutdown 2023-09-30T00:50:46.232 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:46.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 1 -- 172.21.15.40:0/303246784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd80a5fb0 msgr2=0x7f8dd813e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:46.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 1 --2- 172.21.15.40:0/303246784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd80a5fb0 0x7f8dd813e460 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4015000 tx=0x7f8dd4016000 comp rx=0 tx=0).stop 2023-09-30T00:50:46.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 1 -- 172.21.15.40:0/303246784 shutdown_connections 2023-09-30T00:50:46.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 1 --2- 172.21.15.40:0/303246784 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8dd813ac20 0x7f8dd80a7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:46.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 1 --2- 172.21.15.40:0/303246784 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8dd80a5fb0 0x7f8dd813e460 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:46.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 1 --2- 172.21.15.40:0/303246784 >> v2:172.21.15.135:6800/116426 conn(0x7f8dbc068960 0x7f8dbc06ae10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:46.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 1 --2- 172.21.15.40:0/303246784 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8dd80b1460 0x7f8dd80a79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:46.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 1 -- 172.21.15.40:0/303246784 >> 172.21.15.40:0/303246784 conn(0x7f8dd80631f0 msgr2=0x7f8dd80b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:46.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 1 -- 172.21.15.40:0/303246784 shutdown_connections 2023-09-30T00:50:46.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 1 -- 172.21.15.40:0/303246784 wait complete. 2023-09-30T00:50:46.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:46.231+0000 7f8de0129700 1 librados: shutdown 2023-09-30T00:50:49.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:49.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.383+0000 7f31909b7700 10 monclient: get_monmap_and_config 2023-09-30T00:50:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.383+0000 7f31909b7700 10 monclient: build_initial_monmap 2023-09-30T00:50:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.383+0000 7f31909b7700 10 monclient: monmap: 2023-09-30T00:50:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:49.384532+0000 2023-09-30T00:50:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:49.384532+0000 2023-09-30T00:50:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:49.384 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:49.385 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:49.385 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:49.385 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:49.385 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:49.385 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:49.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 1 Processor -- start 2023-09-30T00:50:49.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 1 -- start start 2023-09-30T00:50:49.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 10 monclient: init 2023-09-30T00:50:49.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-09-30T00:50:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f318c0b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:50:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318c0b1830 0x7f318c0a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 10 monclient(hunting): picked mon.noname-b con 0x7f318c0b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f318c13ac20 addr v1:172.21.15.135:6789/0 2023-09-30T00:50:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318c0a5b20 con 0x7f318c0b1460 2023-09-30T00:50:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318c0a71f0 con 0x7f318c13ac20 2023-09-30T00:50:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f318c141e40 con 0x7f318c0b1830 2023-09-30T00:50:49.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.384+0000 7f31909b7700 10 monclient(hunting): authenticate will time out at 5510.685987s 2023-09-30T00:50:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.385+0000 7f318a59c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f318c0b1460 0x7f318c141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:33686/0 (socket says 172.21.15.40:33686) 2023-09-30T00:50:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.385+0000 7f318a59c700 1 -- 172.21.15.40:0/2269341273 learned_addr learned my addr 172.21.15.40:0/2269341273 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:50:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.385+0000 7f3189d9b700 1 --2- 172.21.15.40:0/2269341273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318c0b1830 0x7f318c0a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.386+0000 7f3189d9b700 10 monclient(hunting): get_auth_request con 0x7f318c0b1830 auth_method 0 2023-09-30T00:50:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.386+0000 7f3189d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.386+0000 7f3189d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.386+0000 7f3189d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:49.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.386+0000 7f3189d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.386+0000 7f3189d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.386+0000 7f3189d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.388+0000 7f318959a700 1 -- 172.21.15.40:0/2269341273 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495279617 0 0) 0x7f318c0a5b20 con 0x7f318c0b1460 2023-09-30T00:50:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.388+0000 7f318959a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.388+0000 7f318959a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.388+0000 7f318959a700 10 monclient(hunting): my global_id is 4587 2023-09-30T00:50:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.388+0000 7f318959a700 1 -- 172.21.15.40:0/2269341273 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3174003570 con 0x7f318c0b1460 2023-09-30T00:50:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.389+0000 7f3189d9b700 10 monclient(hunting): handle_auth_done global_id 4523 payload 274 2023-09-30T00:50:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.389+0000 7f3189d9b700 1 -- 172.21.15.40:0/2269341273 >> v1:172.21.15.135:6789/0 conn(0x7f318c13ac20 legacy=0x7f318c13b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.389+0000 7f3189d9b700 1 -- 172.21.15.40:0/2269341273 >> v1:172.21.15.40:6789/0 conn(0x7f318c0b1460 legacy=0x7f318c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.389+0000 7f3189d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:50:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.389+0000 7f3189d9b700 1 monclient: found mon.noname-b 2023-09-30T00:50:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.389+0000 7f3189d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.389+0000 7f3189d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:50:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.389+0000 7f3189d9b700 1 -- 172.21.15.40:0/2269341273 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318c0a74d0 con 0x7f318c0b1830 2023-09-30T00:50:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.391+0000 7f3189d9b700 1 --2- 172.21.15.40:0/2269341273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318c0b1830 0x7f318c0a5fb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3180009910 tx=0x7f31800091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4269515697121139 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:49.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f318959a700 1 -- 172.21.15.40:0/2269341273 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318000b680 con 0x7f318c0b1830 2023-09-30T00:50:49.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f318959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:49.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f318959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:49.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f318959a700 10 monclient: dump: 2023-09-30T00:50:49.396 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:49.396 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:49.396 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:49.396 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:49.396 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:49.397 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f318959a700 10 monclient: _finish_auth 0 2023-09-30T00:50:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f318959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f318959a700 1 -- 172.21.15.40:0/2269341273 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318000b7e0 con 0x7f318c0b1830 2023-09-30T00:50:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f318959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f318959a700 1 -- 172.21.15.40:0/2269341273 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318000bba0 con 0x7f318c0b1830 2023-09-30T00:50:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f31909b7700 5 monclient: authenticate success, global_id 4523 2023-09-30T00:50:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f31909b7700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f31909b7700 4 set_mon_vals no callback set 2023-09-30T00:50:49.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f318959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:49.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f318959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:50:49.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.395+0000 7f318959a700 10 monclient: dump: 2023-09-30T00:50:49.398 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:49.398 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:49.398 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:49.398 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:49.398 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:49.398 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:49.398 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:49.398 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:49.398 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:49.399 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:49.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.396+0000 7f31909b7700 10 monclient: shutdown 2023-09-30T00:50:49.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.396+0000 7f31909b7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:49.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.396+0000 7f31909b7700 1 -- 172.21.15.40:0/2269341273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318c0b1830 msgr2=0x7f318c0a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:49.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.396+0000 7f31909b7700 1 --2- 172.21.15.40:0/2269341273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318c0b1830 0x7f318c0a5fb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3180009910 tx=0x7f31800091a0 comp rx=0 tx=0).stop 2023-09-30T00:50:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.396+0000 7f31909b7700 1 -- 172.21.15.40:0/2269341273 shutdown_connections 2023-09-30T00:50:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.396+0000 7f31909b7700 1 --2- 172.21.15.40:0/2269341273 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318c0b1830 0x7f318c0a5fb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.396+0000 7f31909b7700 1 -- 172.21.15.40:0/2269341273 >> 172.21.15.40:0/2269341273 conn(0x7f318c0631f0 msgr2=0x7f318c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.396+0000 7f31909b7700 1 -- 172.21.15.40:0/2269341273 shutdown_connections 2023-09-30T00:50:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.396+0000 7f31909b7700 1 -- 172.21.15.40:0/2269341273 wait complete. 2023-09-30T00:50:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.397+0000 7f31909b7700 10 monclient: build_initial_monmap 2023-09-30T00:50:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.397+0000 7f31909b7700 10 monclient: monmap: 2023-09-30T00:50:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:49.403 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.397+0000 7f31909b7700 1 librados: starting msgr at 2023-09-30T00:50:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.397+0000 7f31909b7700 1 librados: starting objecter 2023-09-30T00:50:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.397+0000 7f31909b7700 1 Processor -- start 2023-09-30T00:50:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.397+0000 7f31909b7700 1 -- start start 2023-09-30T00:50:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.397+0000 7f31909b7700 1 librados: setting wanted keys 2023-09-30T00:50:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.397+0000 7f31909b7700 1 librados: calling monclient init 2023-09-30T00:50:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.397+0000 7f31909b7700 10 monclient: init 2023-09-30T00:50:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:50:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f318c0b1460 0x7f318c13e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 10 monclient(hunting): picked mon.noname-b con 0x7f318c0b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318c0b1830 0x7f318c0aa180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 10 monclient(hunting): picked mon.noname-c con 0x7f318c0b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f318c13ac20 0x7f318c1514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 10 monclient(hunting): picked mon.noname-a con 0x7f318c13ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318c0aa8e0 con 0x7f318c13ac20 2023-09-30T00:50:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318c0a8c90 con 0x7f318c0b1460 2023-09-30T00:50:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f318c0a98d0 con 0x7f318c0b1830 2023-09-30T00:50:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.398+0000 7f31909b7700 10 monclient(hunting): authenticate will time out at 5510.699606s 2023-09-30T00:50:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f3189d9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318c0b1830 0x7f318c0aa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f318a59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f318c0b1460 0x7f318c13e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f318ad9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f318c13ac20 0x7f318c1514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f3189d9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318c0b1830 0x7f318c0aa180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:35764/0 (socket says 172.21.15.40:35764) 2023-09-30T00:50:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f3189d9b700 1 -- 172.21.15.40:0/15009391 learned_addr learned my addr 172.21.15.40:0/15009391 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f3189d9b700 10 monclient(hunting): get_auth_request con 0x7f318c0b1830 auth_method 0 2023-09-30T00:50:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f3189d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f3189d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f3189d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f318a59c700 10 monclient(hunting): get_auth_request con 0x7f318c0b1460 auth_method 0 2023-09-30T00:50:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f318a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f318a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f318a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f318ad9d700 10 monclient(hunting): get_auth_request con 0x7f318c13ac20 auth_method 0 2023-09-30T00:50:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f318ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f318ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.399+0000 7f318ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.400+0000 7f3189d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.400+0000 7f3189d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.400+0000 7f3189d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.400+0000 7f318a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.400+0000 7f318a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:49.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.400+0000 7f318a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:49.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.400+0000 7f318ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:49.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.400+0000 7f318ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:49.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.400+0000 7f318ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:49.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.403+0000 7f3189d9b700 10 monclient(hunting): handle_auth_done global_id 4526 payload 824 2023-09-30T00:50:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.403+0000 7f3189d9b700 1 -- 172.21.15.40:0/15009391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f318c0b1460 msgr2=0x7f318c13e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.403+0000 7f3189d9b700 1 --2- 172.21.15.40:0/15009391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f318c0b1460 0x7f318c13e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.403+0000 7f3189d9b700 1 -- 172.21.15.40:0/15009391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f318c13ac20 msgr2=0x7f318c1514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.403+0000 7f3189d9b700 1 --2- 172.21.15.40:0/15009391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f318c13ac20 0x7f318c1514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.403+0000 7f3189d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:50:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.403+0000 7f3189d9b700 1 monclient: found mon.noname-c 2023-09-30T00:50:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.403+0000 7f3189d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.403+0000 7f3189d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:50:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.403+0000 7f3189d9b700 1 -- 172.21.15.40:0/15009391 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3180009420 con 0x7f318c0b1830 2023-09-30T00:50:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.406+0000 7f3189d9b700 1 --2- 172.21.15.40:0/15009391 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318c0b1830 0x7f318c0aa180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f318c06f5e0 tx=0x7f3180017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.406+0000 7f317a7fc700 10 client.?.objecter ms_handle_connect 0x7f318c0b1830 2023-09-30T00:50:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.406+0000 7f317a7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:49.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f317a7fc700 1 -- 172.21.15.40:0/15009391 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318000b950 con 0x7f318c0b1830 2023-09-30T00:50:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f317a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f317a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:50:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f317a7fc700 10 monclient: dump: 2023-09-30T00:50:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:49.413 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f317a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:50:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f317a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f317a7fc700 1 -- 172.21.15.40:0/15009391 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318000bad0 con 0x7f318c0b1830 2023-09-30T00:50:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f317a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f317a7fc700 1 -- 172.21.15.40:0/15009391 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318001a680 con 0x7f318c0b1830 2023-09-30T00:50:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f31909b7700 5 monclient: authenticate success, global_id 4526 2023-09-30T00:50:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f31909b7700 10 monclient: _renew_subs 2023-09-30T00:50:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f31909b7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f31909b7700 1 -- 172.21.15.40:0/15009391 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318c0aad70 con 0x7f318c0b1830 2023-09-30T00:50:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f3188d99700 4 set_mon_vals no callback set 2023-09-30T00:50:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f317a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f317a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:50:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f317a7fc700 10 monclient: dump: 2023-09-30T00:50:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:49.414 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:49.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f31909b7700 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f31909b7700 10 monclient: _renew_subs 2023-09-30T00:50:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f31909b7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f31909b7700 1 -- 172.21.15.40:0/15009391 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318c151bd0 con 0x7f318c0b1830 2023-09-30T00:50:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.411+0000 7f31909b7700 1 librados: init done 2023-09-30T00:50:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.412+0000 7f31909b7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.412+0000 7f31909b7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.412+0000 7f31909b7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:49.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.412+0000 7f31909b7700 1 -- 172.21.15.40:0/15009391 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318c064a00 con 0x7f318c0b1830 2023-09-30T00:50:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.432+0000 7f317a7fc700 1 -- 172.21.15.40:0/15009391 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f318001aa70 con 0x7f318c0b1830 2023-09-30T00:50:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.432+0000 7f317a7fc700 1 --2- 172.21.15.40:0/15009391 >> v2:172.21.15.135:6800/116426 conn(0x7f316c068890 0x7f316c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.433+0000 7f318a59c700 1 --2- 172.21.15.40:0/15009391 >> v2:172.21.15.135:6800/116426 conn(0x7f316c068890 0x7f316c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.433+0000 7f318a59c700 10 monclient: get_auth_request con 0x7f316c068890 auth_method 0 2023-09-30T00:50:49.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.433+0000 7f318a59c700 1 --2- 172.21.15.40:0/15009391 >> v2:172.21.15.135:6800/116426 conn(0x7f316c068890 0x7f316c06ad40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f317c007fd0 tx=0x7f317c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:49.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.433+0000 7f317a7fc700 10 client.4526.objecter ms_handle_connect 0x7f316c068890 2023-09-30T00:50:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.438+0000 7f317a7fc700 1 -- 172.21.15.40:0/15009391 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3180057b10 con 0x7f318c0b1830 2023-09-30T00:50:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.438+0000 7f317a7fc700 10 client.4526.objecter ms_dispatch 0x7f318c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.438+0000 7f317a7fc700 3 client.4526.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.438+0000 7f317a7fc700 3 client.4526.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:49.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.438+0000 7f317a7fc700 20 client.4526.objecter dump_active .. 0 homeless 2023-09-30T00:50:49.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.714+0000 7f317a7fc700 1 -- 172.21.15.40:0/15009391 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3180053750 con 0x7f318c0b1830 2023-09-30T00:50:49.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.714+0000 7f317a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:49.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.714+0000 7f317a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:49.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.975+0000 7f31909b7700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:49.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.975+0000 7f31909b7700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:49.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.975+0000 7f31909b7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:49.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.975+0000 7f31909b7700 1 -- 172.21.15.40:0/15009391 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f318c13b590 con 0x7f318c0b1830 2023-09-30T00:50:49.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.983+0000 7f317a7fc700 1 -- 172.21.15.40:0/15009391 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3180056d70 con 0x7f318c0b1830 2023-09-30T00:50:49.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.983+0000 7f317a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:49.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.983+0000 7f317a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:49.983 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:49.986 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 10 librados: watch_flush enter 2023-09-30T00:50:49.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 10 librados: watch_flush exit 2023-09-30T00:50:49.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 20 client.4526.objecter shutdown clearing up homeless session... 2023-09-30T00:50:49.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 10 client.4526.objecter successfully canceled tick 2023-09-30T00:50:49.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 1 -- 172.21.15.40:0/15009391 >> v2:172.21.15.135:6800/116426 conn(0x7f316c068890 msgr2=0x7f316c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:49.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 1 --2- 172.21.15.40:0/15009391 >> v2:172.21.15.135:6800/116426 conn(0x7f316c068890 0x7f316c06ad40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f317c007fd0 tx=0x7f317c009040 comp rx=0 tx=0).stop 2023-09-30T00:50:49.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 10 monclient: shutdown 2023-09-30T00:50:49.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:49.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 1 -- 172.21.15.40:0/15009391 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318c0b1830 msgr2=0x7f318c0aa180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:49.987 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 1 --2- 172.21.15.40:0/15009391 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318c0b1830 0x7f318c0aa180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f318c06f5e0 tx=0x7f3180017000 comp rx=0 tx=0).stop 2023-09-30T00:50:49.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 1 -- 172.21.15.40:0/15009391 shutdown_connections 2023-09-30T00:50:49.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 1 --2- 172.21.15.40:0/15009391 >> v2:172.21.15.135:6800/116426 conn(0x7f316c068890 0x7f316c06ad40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:49.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 1 --2- 172.21.15.40:0/15009391 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f318c0b1460 0x7f318c13e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:49.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 1 --2- 172.21.15.40:0/15009391 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f318c13ac20 0x7f318c1514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:49.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 1 --2- 172.21.15.40:0/15009391 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f318c0b1830 0x7f318c0aa180 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:49.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 1 -- 172.21.15.40:0/15009391 >> 172.21.15.40:0/15009391 conn(0x7f318c0631f0 msgr2=0x7f318c0b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:49.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 1 -- 172.21.15.40:0/15009391 shutdown_connections 2023-09-30T00:50:49.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 1 -- 172.21.15.40:0/15009391 wait complete. 2023-09-30T00:50:49.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:49.985+0000 7f31909b7700 1 librados: shutdown 2023-09-30T00:50:53.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.138+0000 7fec33c7a700 10 monclient: get_monmap_and_config 2023-09-30T00:50:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.138+0000 7fec33c7a700 10 monclient: build_initial_monmap 2023-09-30T00:50:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.138+0000 7fec33c7a700 10 monclient: monmap: 2023-09-30T00:50:53.139 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:53.140 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:53.140 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:53.139710+0000 2023-09-30T00:50:53.140 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:53.139710+0000 2023-09-30T00:50:53.140 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:53.140 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:53.140 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:53.140 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:53.140 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:53.140 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:53.140 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:53.141 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:53.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.139+0000 7fec33c7a700 1 Processor -- start 2023-09-30T00:50:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.139+0000 7fec33c7a700 1 -- start start 2023-09-30T00:50:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.139+0000 7fec33c7a700 10 monclient: init 2023-09-30T00:50:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec33c7a700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec33c7a700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2023-09-30T00:50:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec33c7a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec2c0b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:50:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec33c7a700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fec2c0b1830 0x7fec2c0a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec33c7a700 10 monclient(hunting): picked mon.noname-a con 0x7fec2c0b1830 addr v2:172.21.15.40:3300/0 2023-09-30T00:50:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec33c7a700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fec2c13ac20 0x7fec2c13b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec33c7a700 10 monclient(hunting): picked mon.noname-c con 0x7fec2c13ac20 addr v2:172.21.15.135:3300/0 2023-09-30T00:50:53.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec33c7a700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c0a5b20 con 0x7fec2c0b1460 2023-09-30T00:50:53.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec33c7a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:53.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec33c7a700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec2c141e40 con 0x7fec2c0b1830 2023-09-30T00:50:53.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec33c7a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:53.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec33c7a700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec2c0a64f0 con 0x7fec2c13ac20 2023-09-30T00:50:53.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec33c7a700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:53.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec33c7a700 10 monclient(hunting): authenticate will time out at 5514.441222s 2023-09-30T00:50:53.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.140+0000 7fec31215700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fec2c0b1830 0x7fec2c0a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:53.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.141+0000 7fec32217700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fec2c13ac20 0x7fec2c13b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:53.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.141+0000 7fec31215700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fec2c0b1830 0x7fec2c0a5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59986/0 (socket says 172.21.15.40:59986) 2023-09-30T00:50:53.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.141+0000 7fec31215700 1 -- 172.21.15.40:0/3182498823 learned_addr learned my addr 172.21.15.40:0/3182498823 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:53.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.141+0000 7fec31215700 10 monclient(hunting): get_auth_request con 0x7fec2c0b1830 auth_method 0 2023-09-30T00:50:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.141+0000 7fec31215700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.141+0000 7fec31215700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.141+0000 7fec31215700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.141+0000 7fec32217700 10 monclient(hunting): get_auth_request con 0x7fec2c13ac20 auth_method 0 2023-09-30T00:50:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.141+0000 7fec32217700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.141+0000 7fec32217700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.141+0000 7fec32217700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.142+0000 7fec31215700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.142+0000 7fec31215700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.142+0000 7fec31215700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.142+0000 7fec32217700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:53.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.142+0000 7fec32217700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:53.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.142+0000 7fec32217700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:53.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.144+0000 7fec31215700 10 monclient(hunting): handle_auth_done global_id 4593 payload 274 2023-09-30T00:50:53.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.144+0000 7fec31215700 1 -- 172.21.15.40:0/3182498823 >> v2:172.21.15.135:3300/0 conn(0x7fec2c13ac20 msgr2=0x7fec2c13b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:53.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.144+0000 7fec31215700 1 --2- 172.21.15.40:0/3182498823 >> v2:172.21.15.135:3300/0 conn(0x7fec2c13ac20 0x7fec2c13b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:53.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.144+0000 7fec31215700 1 -- 172.21.15.40:0/3182498823 >> v1:172.21.15.135:6789/0 conn(0x7fec2c0b1460 legacy=0x7fec2c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:53.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.144+0000 7fec31215700 10 monclient: _finish_hunting 0 2023-09-30T00:50:53.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.144+0000 7fec31215700 1 monclient: found mon.noname-a 2023-09-30T00:50:53.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.144+0000 7fec31215700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:53.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.144+0000 7fec31215700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:50:53.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.144+0000 7fec31215700 1 -- 172.21.15.40:0/3182498823 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c0a71f0 con 0x7fec2c0b1830 2023-09-30T00:50:53.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.146+0000 7fec31215700 1 --2- 172.21.15.40:0/3182498823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec2c0b1830 0x7fec2c0a5fb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fec280099e0 tx=0x7fec28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89096b696ee62e2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec30a14700 1 -- 172.21.15.40:0/3182498823 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2800b7e0 con 0x7fec2c0b1830 2023-09-30T00:50:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec30a14700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec30a14700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:50:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec30a14700 10 monclient: dump: 2023-09-30T00:50:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:53.152 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:53.153 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec30a14700 10 monclient: _finish_auth 0 2023-09-30T00:50:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec30a14700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec30a14700 1 -- 172.21.15.40:0/3182498823 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2800b940 con 0x7fec2c0b1830 2023-09-30T00:50:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec30a14700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:53.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec30a14700 1 -- 172.21.15.40:0/3182498823 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2800bce0 con 0x7fec2c0b1830 2023-09-30T00:50:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec33c7a700 5 monclient: authenticate success, global_id 4593 2023-09-30T00:50:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec33c7a700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec30a14700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec33c7a700 4 set_mon_vals no callback set 2023-09-30T00:50:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec30a14700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:50:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.151+0000 7fec30a14700 10 monclient: dump: 2023-09-30T00:50:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:53.154 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:53.155 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:53.155 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:53.155 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:53.155 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:53.155 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.152+0000 7fec33c7a700 10 monclient: shutdown 2023-09-30T00:50:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.152+0000 7fec33c7a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.152+0000 7fec33c7a700 1 -- 172.21.15.40:0/3182498823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec2c0b1830 msgr2=0x7fec2c0a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.152+0000 7fec33c7a700 1 --2- 172.21.15.40:0/3182498823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec2c0b1830 0x7fec2c0a5fb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fec280099e0 tx=0x7fec28009250 comp rx=0 tx=0).stop 2023-09-30T00:50:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.152+0000 7fec33c7a700 1 -- 172.21.15.40:0/3182498823 shutdown_connections 2023-09-30T00:50:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.152+0000 7fec33c7a700 1 --2- 172.21.15.40:0/3182498823 >> v2:172.21.15.135:3300/0 conn(0x7fec2c13ac20 0x7fec2c13b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:53.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.152+0000 7fec33c7a700 1 --2- 172.21.15.40:0/3182498823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec2c0b1830 0x7fec2c0a5fb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.152+0000 7fec33c7a700 1 -- 172.21.15.40:0/3182498823 >> 172.21.15.40:0/3182498823 conn(0x7fec2c0631f0 msgr2=0x7fec2c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.152+0000 7fec33c7a700 1 -- 172.21.15.40:0/3182498823 shutdown_connections 2023-09-30T00:50:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.152+0000 7fec33c7a700 1 -- 172.21.15.40:0/3182498823 wait complete. 2023-09-30T00:50:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.153+0000 7fec33c7a700 10 monclient: build_initial_monmap 2023-09-30T00:50:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.153+0000 7fec33c7a700 10 monclient: monmap: 2023-09-30T00:50:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:53.159 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:53.160 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.153+0000 7fec33c7a700 1 librados: starting msgr at 2023-09-30T00:50:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.153+0000 7fec33c7a700 1 librados: starting objecter 2023-09-30T00:50:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 1 Processor -- start 2023-09-30T00:50:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 1 -- start start 2023-09-30T00:50:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 1 librados: setting wanted keys 2023-09-30T00:50:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 1 librados: calling monclient init 2023-09-30T00:50:53.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 10 monclient: init 2023-09-30T00:50:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:50:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec2c0b1460 0x7fec2c151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 10 monclient(hunting): picked mon.noname-b con 0x7fec2c0b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec2c0b1830 0x7fec2c0a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 10 monclient(hunting): picked mon.noname-c con 0x7fec2c0b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec2c13ac20 0x7fec2c0a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 10 monclient(hunting): picked mon.noname-a con 0x7fec2c13ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c1514e0 con 0x7fec2c13ac20 2023-09-30T00:50:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:53.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c0a7b50 con 0x7fec2c0b1460 2023-09-30T00:50:53.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:53.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c0a8790 con 0x7fec2c0b1830 2023-09-30T00:50:53.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:53.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.154+0000 7fec33c7a700 10 monclient(hunting): authenticate will time out at 5514.455792s 2023-09-30T00:50:53.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.155+0000 7fec31a16700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec2c0b1460 0x7fec2c151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:53.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.155+0000 7fec31215700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec2c0b1830 0x7fec2c0a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:53.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.155+0000 7fec31a16700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec2c0b1460 0x7fec2c151930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:39646/0 (socket says 172.21.15.40:39646) 2023-09-30T00:50:53.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.155+0000 7fec31a16700 1 -- 172.21.15.40:0/438407054 learned_addr learned my addr 172.21.15.40:0/438407054 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:53.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.155+0000 7fec31a16700 10 monclient(hunting): get_auth_request con 0x7fec2c0b1460 auth_method 0 2023-09-30T00:50:53.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.155+0000 7fec31a16700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.155+0000 7fec31a16700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.155+0000 7fec31a16700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec31215700 10 monclient(hunting): get_auth_request con 0x7fec2c0b1830 auth_method 0 2023-09-30T00:50:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec31215700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec31215700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec31215700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec31a16700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec31a16700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec31a16700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec32217700 1 --2- 172.21.15.40:0/438407054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec2c13ac20 0x7fec2c0a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:53.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec31215700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec31215700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec31215700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec32217700 10 monclient(hunting): get_auth_request con 0x7fec2c13ac20 auth_method 0 2023-09-30T00:50:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec32217700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec32217700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.156+0000 7fec32217700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.157+0000 7fec32217700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.157+0000 7fec32217700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:53.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.157+0000 7fec32217700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:53.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.159+0000 7fec31a16700 10 monclient(hunting): handle_auth_done global_id 4651 payload 824 2023-09-30T00:50:53.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.159+0000 7fec31a16700 1 -- 172.21.15.40:0/438407054 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec2c0b1830 msgr2=0x7fec2c0a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:53.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.159+0000 7fec31a16700 1 --2- 172.21.15.40:0/438407054 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec2c0b1830 0x7fec2c0a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:53.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.159+0000 7fec31a16700 1 -- 172.21.15.40:0/438407054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec2c13ac20 msgr2=0x7fec2c0a99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:53.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.159+0000 7fec31a16700 1 --2- 172.21.15.40:0/438407054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec2c13ac20 0x7fec2c0a99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:53.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.159+0000 7fec31a16700 10 monclient: _finish_hunting 0 2023-09-30T00:50:53.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.159+0000 7fec31a16700 1 monclient: found mon.noname-b 2023-09-30T00:50:53.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.160+0000 7fec31a16700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:53.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.160+0000 7fec31a16700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:50:53.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.160+0000 7fec31a16700 1 -- 172.21.15.40:0/438407054 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec280094f0 con 0x7fec2c0b1460 2023-09-30T00:50:53.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.162+0000 7fec31a16700 1 --2- 172.21.15.40:0/438407054 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec2c0b1460 0x7fec2c151930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00b600 tx=0x7fec1c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:53.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.162+0000 7fec21ffb700 10 client.?.objecter ms_handle_connect 0x7fec2c0b1460 2023-09-30T00:50:53.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.162+0000 7fec21ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:53.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec21ffb700 1 -- 172.21.15.40:0/438407054 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c00f510 con 0x7fec2c0b1460 2023-09-30T00:50:53.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:53.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec21ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:53.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec21ffb700 10 monclient: dump: 2023-09-30T00:50:53.169 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:53.169 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:53.169 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:53.169 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:53.169 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:53.169 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:53.169 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:53.170 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:53.170 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:53.170 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:53.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec21ffb700 10 monclient: _finish_auth 0 2023-09-30T00:50:53.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec21ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:53.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec21ffb700 1 -- 172.21.15.40:0/438407054 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1c00f670 con 0x7fec2c0b1460 2023-09-30T00:50:53.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec21ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:53.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec21ffb700 1 -- 172.21.15.40:0/438407054 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c00fa30 con 0x7fec2c0b1460 2023-09-30T00:50:53.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:53.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec21ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:50:53.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec21ffb700 10 monclient: dump: 2023-09-30T00:50:53.171 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:53.171 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:53.171 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:53.171 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:53.171 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:53.171 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:53.171 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:53.171 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:53.171 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:53.171 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:53.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec33c7a700 5 monclient: authenticate success, global_id 4651 2023-09-30T00:50:53.171 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec33c7a700 10 monclient: _renew_subs 2023-09-30T00:50:53.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec33c7a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:53.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec33c7a700 1 -- 172.21.15.40:0/438407054 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c0a89b0 con 0x7fec2c0b1460 2023-09-30T00:50:53.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.167+0000 7fec23fff700 4 set_mon_vals no callback set 2023-09-30T00:50:53.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.168+0000 7fec33c7a700 10 client.4651.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:53.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.168+0000 7fec33c7a700 10 monclient: _renew_subs 2023-09-30T00:50:53.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.168+0000 7fec33c7a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:53.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.168+0000 7fec33c7a700 1 -- 172.21.15.40:0/438407054 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2c13d920 con 0x7fec2c0b1460 2023-09-30T00:50:53.172 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.168+0000 7fec33c7a700 1 librados: init done 2023-09-30T00:50:53.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.169+0000 7fec33c7a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:53.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.169+0000 7fec33c7a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:53.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.169+0000 7fec33c7a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:53.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.169+0000 7fec33c7a700 1 -- 172.21.15.40:0/438407054 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2c064a00 con 0x7fec2c0b1460 2023-09-30T00:50:53.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.419+0000 7fec21ffb700 1 -- 172.21.15.40:0/438407054 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fec1c024430 con 0x7fec2c0b1460 2023-09-30T00:50:53.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.420+0000 7fec21ffb700 1 --2- 172.21.15.40:0/438407054 >> v2:172.21.15.135:6800/116426 conn(0x7fec10068890 0x7fec1006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:53.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.420+0000 7fec31215700 1 --2- 172.21.15.40:0/438407054 >> v2:172.21.15.135:6800/116426 conn(0x7fec10068890 0x7fec1006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:53.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.420+0000 7fec31215700 10 monclient: get_auth_request con 0x7fec10068890 auth_method 0 2023-09-30T00:50:53.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.420+0000 7fec31215700 1 --2- 172.21.15.40:0/438407054 >> v2:172.21.15.135:6800/116426 conn(0x7fec10068890 0x7fec1006ad40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fec280099e0 tx=0x7fec28016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:53.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.421+0000 7fec21ffb700 10 client.4651.objecter ms_handle_connect 0x7fec10068890 2023-09-30T00:50:53.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.425+0000 7fec21ffb700 1 -- 172.21.15.40:0/438407054 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fec1c055fc0 con 0x7fec2c0b1460 2023-09-30T00:50:53.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.425+0000 7fec21ffb700 10 client.4651.objecter ms_dispatch 0x7fec2c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:53.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.425+0000 7fec21ffb700 3 client.4651.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:53.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.425+0000 7fec21ffb700 3 client.4651.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:53.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.425+0000 7fec21ffb700 20 client.4651.objecter dump_active .. 0 homeless 2023-09-30T00:50:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.434+0000 7fec21ffb700 1 -- 172.21.15.40:0/438407054 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fec1c051c00 con 0x7fec2c0b1460 2023-09-30T00:50:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.434+0000 7fec21ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:53.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.434+0000 7fec21ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.716+0000 7fec33c7a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.716+0000 7fec33c7a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.716+0000 7fec33c7a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:50:53.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.716+0000 7fec33c7a700 1 -- 172.21.15.40:0/438407054 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fec2c0afca0 con 0x7fec2c0b1460 2023-09-30T00:50:53.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.721+0000 7fec21ffb700 1 -- 172.21.15.40:0/438407054 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fec1c055220 con 0x7fec2c0b1460 2023-09-30T00:50:53.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.721+0000 7fec21ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:53.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.721+0000 7fec21ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:53.722 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:53.722 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 10 librados: watch_flush enter 2023-09-30T00:50:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 10 librados: watch_flush exit 2023-09-30T00:50:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 20 client.4651.objecter shutdown clearing up homeless session... 2023-09-30T00:50:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 10 client.4651.objecter successfully canceled tick 2023-09-30T00:50:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 1 -- 172.21.15.40:0/438407054 >> v2:172.21.15.135:6800/116426 conn(0x7fec10068890 msgr2=0x7fec1006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:53.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 1 --2- 172.21.15.40:0/438407054 >> v2:172.21.15.135:6800/116426 conn(0x7fec10068890 0x7fec1006ad40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fec280099e0 tx=0x7fec28016000 comp rx=0 tx=0).stop 2023-09-30T00:50:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 10 monclient: shutdown 2023-09-30T00:50:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 1 -- 172.21.15.40:0/438407054 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec2c0b1460 msgr2=0x7fec2c151930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 1 --2- 172.21.15.40:0/438407054 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec2c0b1460 0x7fec2c151930 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00b600 tx=0x7fec1c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:50:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 1 -- 172.21.15.40:0/438407054 shutdown_connections 2023-09-30T00:50:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 1 --2- 172.21.15.40:0/438407054 >> v2:172.21.15.135:6800/116426 conn(0x7fec10068890 0x7fec1006ad40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 1 --2- 172.21.15.40:0/438407054 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec2c0b1460 0x7fec2c151930 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 1 --2- 172.21.15.40:0/438407054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec2c13ac20 0x7fec2c0a99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 1 --2- 172.21.15.40:0/438407054 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec2c0b1830 0x7fec2c0a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 1 -- 172.21.15.40:0/438407054 >> 172.21.15.40:0/438407054 conn(0x7fec2c0631f0 msgr2=0x7fec2c139d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:53.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 1 -- 172.21.15.40:0/438407054 shutdown_connections 2023-09-30T00:50:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 1 -- 172.21.15.40:0/438407054 wait complete. 2023-09-30T00:50:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:53.724+0000 7fec33c7a700 1 librados: shutdown 2023-09-30T00:50:56.739 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:50:56.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.882+0000 7f1ff33cc700 10 monclient: get_monmap_and_config 2023-09-30T00:50:56.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.882+0000 7f1ff33cc700 10 monclient: build_initial_monmap 2023-09-30T00:50:56.883 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.882+0000 7f1ff33cc700 10 monclient: monmap: 2023-09-30T00:50:56.883 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:56.883 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:56.883 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:50:56.883672+0000 2023-09-30T00:50:56.883 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:50:56.883672+0000 2023-09-30T00:50:56.883 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:56.883 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:56.884 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:50:56.884 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:50:56.884 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:50:56.884 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:50:56.884 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:50:56.884 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.883+0000 7f1ff33cc700 1 Processor -- start 2023-09-30T00:50:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.883+0000 7f1ff33cc700 1 -- start start 2023-09-30T00:50:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.883+0000 7f1ff33cc700 10 monclient: init 2023-09-30T00:50:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.884+0000 7f1ff33cc700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.884+0000 7f1ff33cc700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-09-30T00:50:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.884+0000 7f1ff33cc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1fec0af4f0 addr v1:172.21.15.135:6789/0 2023-09-30T00:50:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.884+0000 7f1ff33cc700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1fec0ad210 0x7f1fec06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.884+0000 7f1ff33cc700 10 monclient(hunting): picked mon.noname-c con 0x7f1fec0ad210 addr v2:172.21.15.135:3300/0 2023-09-30T00:50:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.884+0000 7f1ff33cc700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1fec141ec0 0x7f1fec13b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.884+0000 7f1ff33cc700 10 monclient(hunting): picked mon.noname-a con 0x7f1fec141ec0 addr v2:172.21.15.40:3300/0 2023-09-30T00:50:56.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.884+0000 7f1ff33cc700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fec142340 con 0x7f1fec0af4f0 2023-09-30T00:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.884+0000 7f1ff33cc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.884+0000 7f1ff33cc700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fec0add50 con 0x7f1fec141ec0 2023-09-30T00:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.884+0000 7f1ff33cc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.884+0000 7f1ff33cc700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fec0ad8c0 con 0x7f1fec0ad210 2023-09-30T00:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.884+0000 7f1ff33cc700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.884+0000 7f1ff33cc700 10 monclient(hunting): authenticate will time out at 5518.185391s 2023-09-30T00:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.885+0000 7f1ff1969700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1fec141ec0 0x7f1fec13b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.885+0000 7f1ff1969700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1fec141ec0 0x7f1fec13b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:59996/0 (socket says 172.21.15.40:59996) 2023-09-30T00:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.885+0000 7f1ff0967700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f1fec0ad210 0x7f1fec06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.885+0000 7f1ff1969700 1 -- 172.21.15.40:0/2715195786 learned_addr learned my addr 172.21.15.40:0/2715195786 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.885+0000 7f1ff1969700 10 monclient(hunting): get_auth_request con 0x7f1fec141ec0 auth_method 0 2023-09-30T00:50:56.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.885+0000 7f1ff1969700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.885+0000 7f1ff1969700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.885+0000 7f1ff1969700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:56.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.886+0000 7f1ff0967700 10 monclient(hunting): get_auth_request con 0x7f1fec0ad210 auth_method 0 2023-09-30T00:50:56.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.886+0000 7f1ff0967700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:56.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.886+0000 7f1ff0967700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:56.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.886+0000 7f1ff0967700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:56.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.886+0000 7f1ff1969700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:56.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.886+0000 7f1ff1969700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:56.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.886+0000 7f1ff1969700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:56.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.886+0000 7f1ff0967700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:56.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.886+0000 7f1ff0967700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:56.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.886+0000 7f1ff0967700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:56.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.888+0000 7f1ff1969700 10 monclient(hunting): handle_auth_done global_id 4599 payload 274 2023-09-30T00:50:56.889 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.888+0000 7f1ff1969700 1 -- 172.21.15.40:0/2715195786 >> v2:172.21.15.135:3300/0 conn(0x7f1fec0ad210 msgr2=0x7f1fec06ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:56.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.888+0000 7f1ff1969700 1 --2- 172.21.15.40:0/2715195786 >> v2:172.21.15.135:3300/0 conn(0x7f1fec0ad210 0x7f1fec06ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:56.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.888+0000 7f1ff1969700 1 -- 172.21.15.40:0/2715195786 >> v1:172.21.15.135:6789/0 conn(0x7f1fec0af4f0 legacy=0x7f1fec1417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:56.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.888+0000 7f1ff1969700 10 monclient: _finish_hunting 0 2023-09-30T00:50:56.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.888+0000 7f1ff1969700 1 monclient: found mon.noname-a 2023-09-30T00:50:56.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.888+0000 7f1ff1969700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:56.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.888+0000 7f1ff1969700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:50:56.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.888+0000 7f1ff1969700 1 -- 172.21.15.40:0/2715195786 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec0af8c0 con 0x7f1fec141ec0 2023-09-30T00:50:56.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.891+0000 7f1ff1969700 1 --2- 172.21.15.40:0/2715195786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec141ec0 0x7f1fec13b010 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1fec064190 tx=0x7f1fdc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a3f5749202494c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1febfff700 1 -- 172.21.15.40:0/2715195786 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc00e070 con 0x7f1fec141ec0 2023-09-30T00:50:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1febfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1febfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:50:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1febfff700 10 monclient: dump: 2023-09-30T00:50:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:56.896 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:56.897 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1febfff700 10 monclient: _finish_auth 0 2023-09-30T00:50:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1febfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1febfff700 1 -- 172.21.15.40:0/2715195786 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc00b9e0 con 0x7f1fec141ec0 2023-09-30T00:50:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1febfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1febfff700 1 -- 172.21.15.40:0/2715195786 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0092e0 con 0x7f1fec141ec0 2023-09-30T00:50:56.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1febfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1febfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:50:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1febfff700 10 monclient: dump: 2023-09-30T00:50:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:56.898 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:56.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1ff33cc700 5 monclient: authenticate success, global_id 4599 2023-09-30T00:50:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1ff33cc700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:56.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.895+0000 7f1ff33cc700 4 set_mon_vals no callback set 2023-09-30T00:50:56.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.896+0000 7f1ff33cc700 10 monclient: shutdown 2023-09-30T00:50:56.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.896+0000 7f1ff33cc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:56.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.896+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2715195786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec141ec0 msgr2=0x7f1fec13b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:56.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.896+0000 7f1ff33cc700 1 --2- 172.21.15.40:0/2715195786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec141ec0 0x7f1fec13b010 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1fec064190 tx=0x7f1fdc009840 comp rx=0 tx=0).stop 2023-09-30T00:50:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.896+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2715195786 shutdown_connections 2023-09-30T00:50:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.896+0000 7f1ff33cc700 1 --2- 172.21.15.40:0/2715195786 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec141ec0 0x7f1fec13b010 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.896+0000 7f1ff33cc700 1 --2- 172.21.15.40:0/2715195786 >> v2:172.21.15.135:3300/0 conn(0x7f1fec0ad210 0x7f1fec06ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.896+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2715195786 >> 172.21.15.40:0/2715195786 conn(0x7f1fec06af20 msgr2=0x7f1fec0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.896+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2715195786 shutdown_connections 2023-09-30T00:50:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.896+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2715195786 wait complete. 2023-09-30T00:50:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.897+0000 7f1ff33cc700 10 monclient: build_initial_monmap 2023-09-30T00:50:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.897+0000 7f1ff33cc700 10 monclient: monmap: 2023-09-30T00:50:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:50:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:50:56.902 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:50:56.903 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:56.903 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:50:56.903 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:50:56.903 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:50:56.903 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:50:56.903 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:56.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.897+0000 7f1ff33cc700 1 librados: starting msgr at 2023-09-30T00:50:56.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.897+0000 7f1ff33cc700 1 librados: starting objecter 2023-09-30T00:50:56.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.897+0000 7f1ff33cc700 1 Processor -- start 2023-09-30T00:50:56.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.897+0000 7f1ff33cc700 1 -- start start 2023-09-30T00:50:56.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.897+0000 7f1ff33cc700 1 librados: setting wanted keys 2023-09-30T00:50:56.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.897+0000 7f1ff33cc700 1 librados: calling monclient init 2023-09-30T00:50:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.897+0000 7f1ff33cc700 10 monclient: init 2023-09-30T00:50:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:50:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fec0ad210 0x7f1fec13ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 10 monclient(hunting): picked mon.noname-c con 0x7f1fec0ad210 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:50:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1fec0af4f0 0x7f1fec13ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 10 monclient(hunting): picked mon.noname-b con 0x7f1fec0af4f0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:50:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec141ec0 0x7f1fec151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 10 monclient(hunting): picked mon.noname-a con 0x7f1fec141ec0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:50:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec13d6b0 con 0x7f1fec141ec0 2023-09-30T00:50:56.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:56.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec0a99f0 con 0x7f1fec0af4f0 2023-09-30T00:50:56.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:56.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fec13e260 con 0x7f1fec0ad210 2023-09-30T00:50:56.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:56.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.898+0000 7f1ff33cc700 10 monclient(hunting): authenticate will time out at 5518.199743s 2023-09-30T00:50:56.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff1168700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fec0ad210 0x7f1fec13ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:56.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff1969700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec141ec0 0x7f1fec151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:56.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff0967700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1fec0af4f0 0x7f1fec13ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:56.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff1969700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec141ec0 0x7f1fec151230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60010/0 (socket says 172.21.15.40:60010) 2023-09-30T00:50:56.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff1969700 1 -- 172.21.15.40:0/2988154672 learned_addr learned my addr 172.21.15.40:0/2988154672 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:50:56.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff1969700 10 monclient(hunting): get_auth_request con 0x7f1fec141ec0 auth_method 0 2023-09-30T00:50:56.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff1969700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:56.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff1969700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:56.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff1969700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:56.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff1168700 10 monclient(hunting): get_auth_request con 0x7f1fec0ad210 auth_method 0 2023-09-30T00:50:56.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff1168700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:56.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff1168700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:56.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff1168700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:56.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff0967700 10 monclient(hunting): get_auth_request con 0x7f1fec0af4f0 auth_method 0 2023-09-30T00:50:56.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff0967700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:56.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff0967700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:56.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.899+0000 7f1ff0967700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:56.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.900+0000 7f1ff1969700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:56.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.900+0000 7f1ff1969700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:56.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.900+0000 7f1ff1969700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:56.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.900+0000 7f1ff1168700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:56.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.900+0000 7f1ff1168700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:56.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.900+0000 7f1ff1168700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:56.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.900+0000 7f1ff0967700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:56.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.900+0000 7f1ff0967700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:56.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.900+0000 7f1ff0967700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:56.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.904+0000 7f1ff1168700 10 monclient(hunting): handle_auth_done global_id 4532 payload 824 2023-09-30T00:50:56.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.904+0000 7f1ff1168700 1 -- 172.21.15.40:0/2988154672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1fec0af4f0 msgr2=0x7f1fec13ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:56.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.904+0000 7f1ff1168700 1 --2- 172.21.15.40:0/2988154672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1fec0af4f0 0x7f1fec13ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:56.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.904+0000 7f1ff1168700 1 -- 172.21.15.40:0/2988154672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec141ec0 msgr2=0x7f1fec151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:56.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.904+0000 7f1ff1168700 1 --2- 172.21.15.40:0/2988154672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec141ec0 0x7f1fec151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:56.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.904+0000 7f1ff1168700 10 monclient: _finish_hunting 0 2023-09-30T00:50:56.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.904+0000 7f1ff1168700 1 monclient: found mon.noname-c 2023-09-30T00:50:56.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.904+0000 7f1ff1168700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:56.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.904+0000 7f1ff1168700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:50:56.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.904+0000 7f1ff1168700 1 -- 172.21.15.40:0/2988154672 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc009ae0 con 0x7f1fec0ad210 2023-09-30T00:50:56.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.908+0000 7f1ff1168700 1 --2- 172.21.15.40:0/2988154672 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fec0ad210 0x7f1fec13ed60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400b5b0 tx=0x7f1fe40090f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:56.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.908+0000 7f1fe97fa700 10 client.?.objecter ms_handle_connect 0x7f1fec0ad210 2023-09-30T00:50:56.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.908+0000 7f1fe97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:50:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1fe97fa700 1 -- 172.21.15.40:0/2988154672 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe400f510 con 0x7f1fec0ad210 2023-09-30T00:50:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1fe97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1fe97fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:50:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1fe97fa700 10 monclient: dump: 2023-09-30T00:50:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:56.915 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:56.916 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1fe97fa700 10 monclient: _finish_auth 0 2023-09-30T00:50:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1fe97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1fe97fa700 1 -- 172.21.15.40:0/2988154672 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe400f6b0 con 0x7f1fec0ad210 2023-09-30T00:50:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1fe97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1fe97fa700 1 -- 172.21.15.40:0/2988154672 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe400fa70 con 0x7f1fec0ad210 2023-09-30T00:50:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1fe97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:56.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1fe97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:50:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1fe97fa700 10 monclient: dump: 2023-09-30T00:50:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:50:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:50:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:50:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:50:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:50:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:50:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:50:56.917 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:50:56.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1ff33cc700 5 monclient: authenticate success, global_id 4532 2023-09-30T00:50:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1ff33cc700 10 monclient: _renew_subs 2023-09-30T00:50:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1ff33cc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2988154672 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fec13e540 con 0x7f1fec0ad210 2023-09-30T00:50:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1feb7fe700 4 set_mon_vals no callback set 2023-09-30T00:50:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1ff33cc700 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:50:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1ff33cc700 10 monclient: _renew_subs 2023-09-30T00:50:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1ff33cc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2988154672 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fec1519b0 con 0x7f1fec0ad210 2023-09-30T00:50:56.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.914+0000 7f1ff33cc700 1 librados: init done 2023-09-30T00:50:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.915+0000 7f1ff33cc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.915+0000 7f1ff33cc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.915+0000 7f1ff33cc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:56.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:56.915+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2988154672 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fec0a9c10 con 0x7f1fec0ad210 2023-09-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.218+0000 7f1fe97fa700 1 -- 172.21.15.40:0/2988154672 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1fe4024430 con 0x7f1fec0ad210 2023-09-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.219+0000 7f1fe97fa700 1 --2- 172.21.15.40:0/2988154672 >> v2:172.21.15.135:6800/116426 conn(0x7f1fd8068960 0x7f1fd806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.219+0000 7f1fe97fa700 1 -- 172.21.15.40:0/2988154672 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1fe4083ee0 con 0x7f1fec0ad210 2023-09-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.219+0000 7f1fe97fa700 10 client.4532.objecter ms_dispatch 0x7f1fec001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.219+0000 7f1fe97fa700 3 client.4532.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.219+0000 7f1fe97fa700 3 client.4532.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.219+0000 7f1fe97fa700 20 client.4532.objecter dump_active .. 0 homeless 2023-09-30T00:50:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.219+0000 7f1fe97fa700 1 -- 172.21.15.40:0/2988154672 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1fe40843e0 con 0x7f1fec0ad210 2023-09-30T00:50:57.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.219+0000 7f1fe97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:57.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.219+0000 7f1fe97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:57.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.219+0000 7f1ff0967700 1 --2- 172.21.15.40:0/2988154672 >> v2:172.21.15.135:6800/116426 conn(0x7f1fd8068960 0x7f1fd806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:57.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.219+0000 7f1ff0967700 10 monclient: get_auth_request con 0x7f1fd8068960 auth_method 0 2023-09-30T00:50:57.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.220+0000 7f1fe97fa700 10 client.4532.objecter ms_handle_connect 0x7f1fd8068960 2023-09-30T00:50:57.221 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.220+0000 7f1ff0967700 1 --2- 172.21.15.40:0/2988154672 >> v2:172.21.15.135:6800/116426 conn(0x7f1fd8068960 0x7f1fd806ae10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0aab90 tx=0x7f1fe00091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:57.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.478+0000 7f1ff33cc700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:50:57.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.478+0000 7f1ff33cc700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:57.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.478+0000 7f1ff33cc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:50:57.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.478+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2988154672 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1fec064a00 con 0x7f1fec0ad210 2023-09-30T00:50:57.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.486+0000 7f1fe97fa700 1 -- 172.21.15.40:0/2988154672 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1fe40519b0 con 0x7f1fec0ad210 2023-09-30T00:50:57.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.486+0000 7f1fe97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:50:57.487 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.486+0000 7f1fe97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:57.487 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:50:57.487 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:50:57.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 10 librados: watch_flush enter 2023-09-30T00:50:57.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 10 librados: watch_flush exit 2023-09-30T00:50:57.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 20 client.4532.objecter shutdown clearing up homeless session... 2023-09-30T00:50:57.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 10 client.4532.objecter successfully canceled tick 2023-09-30T00:50:57.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2988154672 >> v2:172.21.15.135:6800/116426 conn(0x7f1fd8068960 msgr2=0x7f1fd806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:57.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 1 --2- 172.21.15.40:0/2988154672 >> v2:172.21.15.135:6800/116426 conn(0x7f1fd8068960 0x7f1fd806ae10 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1fec0aab90 tx=0x7f1fe00091a0 comp rx=0 tx=0).stop 2023-09-30T00:50:57.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 10 monclient: shutdown 2023-09-30T00:50:57.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:57.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2988154672 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fec0ad210 msgr2=0x7f1fec13ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:57.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 1 --2- 172.21.15.40:0/2988154672 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fec0ad210 0x7f1fec13ed60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400b5b0 tx=0x7f1fe40090f0 comp rx=0 tx=0).stop 2023-09-30T00:50:57.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2988154672 shutdown_connections 2023-09-30T00:50:57.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 1 --2- 172.21.15.40:0/2988154672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fec141ec0 0x7f1fec151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:57.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 1 --2- 172.21.15.40:0/2988154672 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f1fec0ad210 0x7f1fec13ed60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:57.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 1 --2- 172.21.15.40:0/2988154672 >> v2:172.21.15.135:6800/116426 conn(0x7f1fd8068960 0x7f1fd806ae10 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:57.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 1 --2- 172.21.15.40:0/2988154672 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f1fec0af4f0 0x7f1fec13ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:57.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2988154672 >> 172.21.15.40:0/2988154672 conn(0x7f1fec06af20 msgr2=0x7f1fec0b2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:57.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2988154672 shutdown_connections 2023-09-30T00:50:57.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 1 -- 172.21.15.40:0/2988154672 wait complete. 2023-09-30T00:50:57.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:50:57.489+0000 7f1ff33cc700 1 librados: shutdown 2023-09-30T00:51:00.504 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:00.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.652+0000 7efc6dad8700 10 monclient: get_monmap_and_config 2023-09-30T00:51:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.652+0000 7efc6dad8700 10 monclient: build_initial_monmap 2023-09-30T00:51:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.652+0000 7efc6dad8700 10 monclient: monmap: 2023-09-30T00:51:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:00.652110+0000 2023-09-30T00:51:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:00.652110+0000 2023-09-30T00:51:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:00.652 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:00.653 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.652+0000 7efc6dad8700 1 Processor -- start 2023-09-30T00:51:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 1 -- start start 2023-09-30T00:51:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 10 monclient: init 2023-09-30T00:51:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-09-30T00:51:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efc680b1460 0x7efc680b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 10 monclient(hunting): picked mon.noname-c con 0x7efc680b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:51:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc680a5fb0 0x7efc68141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 10 monclient(hunting): picked mon.noname-b con 0x7efc680a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efc680af410 0x7efc680af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 10 monclient(hunting): picked mon.noname-a con 0x7efc680af410 addr v2:172.21.15.40:3300/0 2023-09-30T00:51:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc680a71f0 con 0x7efc680af410 2023-09-30T00:51:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:00.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc680a74d0 con 0x7efc680b1460 2023-09-30T00:51:00.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:00.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc680a63c0 con 0x7efc680a5fb0 2023-09-30T00:51:00.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:00.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.653+0000 7efc6dad8700 10 monclient(hunting): authenticate will time out at 5521.953760s 2023-09-30T00:51:00.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.654+0000 7efc677fe700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efc680b1460 0x7efc680b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:00.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.655+0000 7efc677fe700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7efc680b1460 0x7efc680b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:39666/0 (socket says 172.21.15.40:39666) 2023-09-30T00:51:00.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.655+0000 7efc677fe700 1 -- 172.21.15.40:0/2929382394 learned_addr learned my addr 172.21.15.40:0/2929382394 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:00.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.655+0000 7efc677fe700 10 monclient(hunting): get_auth_request con 0x7efc680b1460 auth_method 0 2023-09-30T00:51:00.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.655+0000 7efc677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:00.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.655+0000 7efc677fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:00.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.655+0000 7efc677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:00.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.655+0000 7efc67fff700 1 --2- 172.21.15.40:0/2929382394 >> v2:172.21.15.40:3300/0 conn(0x7efc680af410 0x7efc680af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.655+0000 7efc677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.655+0000 7efc677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.655+0000 7efc677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.656+0000 7efc66ffd700 1 --2- 172.21.15.40:0/2929382394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc680a5fb0 0x7efc68141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.656+0000 7efc66ffd700 10 monclient(hunting): get_auth_request con 0x7efc680a5fb0 auth_method 0 2023-09-30T00:51:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.656+0000 7efc66ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.656+0000 7efc66ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:00.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.656+0000 7efc66ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.657+0000 7efc66ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.657+0000 7efc66ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.657+0000 7efc66ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.657+0000 7efc677fe700 10 monclient(hunting): handle_auth_done global_id 4660 payload 274 2023-09-30T00:51:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.657+0000 7efc677fe700 1 -- 172.21.15.40:0/2929382394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc680a5fb0 msgr2=0x7efc68141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.657+0000 7efc677fe700 1 --2- 172.21.15.40:0/2929382394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc680a5fb0 0x7efc68141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.657+0000 7efc677fe700 1 -- 172.21.15.40:0/2929382394 >> v2:172.21.15.40:3300/0 conn(0x7efc680af410 msgr2=0x7efc680af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.657+0000 7efc677fe700 1 --2- 172.21.15.40:0/2929382394 >> v2:172.21.15.40:3300/0 conn(0x7efc680af410 0x7efc680af890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.658+0000 7efc677fe700 10 monclient: _finish_hunting 0 2023-09-30T00:51:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.658+0000 7efc677fe700 1 monclient: found mon.noname-c 2023-09-30T00:51:00.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.658+0000 7efc677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.658+0000 7efc677fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:51:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.658+0000 7efc677fe700 1 -- 172.21.15.40:0/2929382394 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc680a56d0 con 0x7efc680b1460 2023-09-30T00:51:00.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.660+0000 7efc677fe700 1 --2- 172.21.15.40:0/2929382394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc680b1460 0x7efc680b1830 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efc680b1f90 tx=0x7efc58009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35bc0447be8acf3b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc667fc700 1 -- 172.21.15.40:0/2929382394 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5800e070 con 0x7efc680b1460 2023-09-30T00:51:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc667fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:51:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc667fc700 10 monclient: dump: 2023-09-30T00:51:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:00.664 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:00.665 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc667fc700 10 monclient: _finish_auth 0 2023-09-30T00:51:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc667fc700 1 -- 172.21.15.40:0/2929382394 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5800ba20 con 0x7efc680b1460 2023-09-30T00:51:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc667fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc667fc700 1 -- 172.21.15.40:0/2929382394 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc580092e0 con 0x7efc680b1460 2023-09-30T00:51:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:00.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc667fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc667fc700 10 monclient: dump: 2023-09-30T00:51:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:00.666 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:00.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc6dad8700 5 monclient: authenticate success, global_id 4660 2023-09-30T00:51:00.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc6dad8700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:00.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.664+0000 7efc6dad8700 4 set_mon_vals no callback set 2023-09-30T00:51:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.665+0000 7efc6dad8700 10 monclient: shutdown 2023-09-30T00:51:00.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.665+0000 7efc6dad8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.665+0000 7efc6dad8700 1 -- 172.21.15.40:0/2929382394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc680b1460 msgr2=0x7efc680b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.665+0000 7efc6dad8700 1 --2- 172.21.15.40:0/2929382394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc680b1460 0x7efc680b1830 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efc680b1f90 tx=0x7efc58009840 comp rx=0 tx=0).stop 2023-09-30T00:51:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.665+0000 7efc6dad8700 1 -- 172.21.15.40:0/2929382394 shutdown_connections 2023-09-30T00:51:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.665+0000 7efc6dad8700 1 --2- 172.21.15.40:0/2929382394 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc680b1460 0x7efc680b1830 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:00.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.665+0000 7efc6dad8700 1 --2- 172.21.15.40:0/2929382394 >> v2:172.21.15.40:3300/0 conn(0x7efc680af410 0x7efc680af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.665+0000 7efc6dad8700 1 --2- 172.21.15.40:0/2929382394 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc680a5fb0 0x7efc68141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.665+0000 7efc6dad8700 1 -- 172.21.15.40:0/2929382394 >> 172.21.15.40:0/2929382394 conn(0x7efc680631f0 msgr2=0x7efc680635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.665+0000 7efc6dad8700 1 -- 172.21.15.40:0/2929382394 shutdown_connections 2023-09-30T00:51:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.665+0000 7efc6dad8700 1 -- 172.21.15.40:0/2929382394 wait complete. 2023-09-30T00:51:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.666+0000 7efc6dad8700 10 monclient: build_initial_monmap 2023-09-30T00:51:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.666+0000 7efc6dad8700 10 monclient: monmap: 2023-09-30T00:51:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:00.675 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.666+0000 7efc6dad8700 1 librados: starting msgr at 2023-09-30T00:51:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.666+0000 7efc6dad8700 1 librados: starting objecter 2023-09-30T00:51:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.667+0000 7efc6dad8700 1 Processor -- start 2023-09-30T00:51:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.667+0000 7efc6dad8700 1 -- start start 2023-09-30T00:51:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.667+0000 7efc6dad8700 1 librados: setting wanted keys 2023-09-30T00:51:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.667+0000 7efc6dad8700 1 librados: calling monclient init 2023-09-30T00:51:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.667+0000 7efc6dad8700 10 monclient: init 2023-09-30T00:51:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:51:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc680a5fb0 0x7efc6813dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:00.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 10 monclient(hunting): picked mon.noname-a con 0x7efc680a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:00.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc680af410 0x7efc680ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:00.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 10 monclient(hunting): picked mon.noname-b con 0x7efc680af410 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:00.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc680b1460 0x7efc680a79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:00.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 10 monclient(hunting): picked mon.noname-c con 0x7efc680b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:00.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:00.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc680a9e00 con 0x7efc680a5fb0 2023-09-30T00:51:00.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:00.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc680a9b00 con 0x7efc680af410 2023-09-30T00:51:00.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:00.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc680ab650 con 0x7efc680b1460 2023-09-30T00:51:00.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:00.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc6dad8700 10 monclient(hunting): authenticate will time out at 5521.968378s 2023-09-30T00:51:00.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc677fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc680a5fb0 0x7efc6813dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:00.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc67fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc680b1460 0x7efc680a79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:00.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.668+0000 7efc66ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc680af410 0x7efc680ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:00.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc67fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc680b1460 0x7efc680a79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:54096/0 (socket says 172.21.15.40:54096) 2023-09-30T00:51:00.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc67fff700 1 -- 172.21.15.40:0/2249605654 learned_addr learned my addr 172.21.15.40:0/2249605654 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:00.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc67fff700 10 monclient(hunting): get_auth_request con 0x7efc680b1460 auth_method 0 2023-09-30T00:51:00.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:00.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc67fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:00.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc67fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:00.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc66ffd700 10 monclient(hunting): get_auth_request con 0x7efc680af410 auth_method 0 2023-09-30T00:51:00.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc66ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc66ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc66ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc677fe700 10 monclient(hunting): get_auth_request con 0x7efc680a5fb0 auth_method 0 2023-09-30T00:51:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc677fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc67fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc67fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc67fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc66ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc66ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.669+0000 7efc66ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:00.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.670+0000 7efc677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:00.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.670+0000 7efc677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:00.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.670+0000 7efc677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:00.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.672+0000 7efc66ffd700 10 monclient(hunting): handle_auth_done global_id 4663 payload 824 2023-09-30T00:51:00.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.673+0000 7efc66ffd700 1 -- 172.21.15.40:0/2249605654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc680b1460 msgr2=0x7efc680a79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:00.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.673+0000 7efc66ffd700 1 --2- 172.21.15.40:0/2249605654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc680b1460 0x7efc680a79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:00.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.673+0000 7efc66ffd700 1 -- 172.21.15.40:0/2249605654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc680a5fb0 msgr2=0x7efc6813dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:00.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.673+0000 7efc66ffd700 1 --2- 172.21.15.40:0/2249605654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc680a5fb0 0x7efc6813dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:00.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.673+0000 7efc66ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:51:00.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.673+0000 7efc66ffd700 1 monclient: found mon.noname-b 2023-09-30T00:51:00.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.673+0000 7efc66ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:00.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.673+0000 7efc66ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:51:00.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.673+0000 7efc66ffd700 1 -- 172.21.15.40:0/2249605654 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc58009ae0 con 0x7efc680af410 2023-09-30T00:51:00.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.675+0000 7efc66ffd700 1 --2- 172.21.15.40:0/2249605654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc680af410 0x7efc680ae0d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efc5c00ea50 tx=0x7efc5c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.675+0000 7efc4f7fe700 10 client.?.objecter ms_handle_connect 0x7efc680af410 2023-09-30T00:51:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.675+0000 7efc4f7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc4f7fe700 1 -- 172.21.15.40:0/2249605654 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c00ee90 con 0x7efc680af410 2023-09-30T00:51:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc4f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc4f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc4f7fe700 10 monclient: dump: 2023-09-30T00:51:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:00.683 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc4f7fe700 10 monclient: _finish_auth 0 2023-09-30T00:51:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc4f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc4f7fe700 1 -- 172.21.15.40:0/2249605654 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5c00c7f0 con 0x7efc680af410 2023-09-30T00:51:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc6dad8700 5 monclient: authenticate success, global_id 4663 2023-09-30T00:51:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc6dad8700 10 monclient: _renew_subs 2023-09-30T00:51:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc6dad8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc6dad8700 1 -- 172.21.15.40:0/2249605654 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc6813e2e0 con 0x7efc680af410 2023-09-30T00:51:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc4f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc4f7fe700 1 -- 172.21.15.40:0/2249605654 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5c00cb90 con 0x7efc680af410 2023-09-30T00:51:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc4f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc4f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc4f7fe700 10 monclient: dump: 2023-09-30T00:51:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:00.685 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:00.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc65ffb700 4 set_mon_vals no callback set 2023-09-30T00:51:00.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc6dad8700 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:00.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc6dad8700 10 monclient: _renew_subs 2023-09-30T00:51:00.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc6dad8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:00.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc6dad8700 1 -- 172.21.15.40:0/2249605654 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc68139ee0 con 0x7efc680af410 2023-09-30T00:51:00.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.679+0000 7efc6dad8700 1 librados: init done 2023-09-30T00:51:00.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.680+0000 7efc6dad8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:00.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.680+0000 7efc6dad8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:00.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.680+0000 7efc6dad8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:00.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.680+0000 7efc6dad8700 1 -- 172.21.15.40:0/2249605654 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc680a9fb0 con 0x7efc680af410 2023-09-30T00:51:00.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.699+0000 7efc4f7fe700 1 -- 172.21.15.40:0/2249605654 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7efc5c013470 con 0x7efc680af410 2023-09-30T00:51:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.699+0000 7efc4f7fe700 1 --2- 172.21.15.40:0/2249605654 >> v2:172.21.15.135:6800/116426 conn(0x7efc54068890 0x7efc5406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.699+0000 7efc677fe700 1 --2- 172.21.15.40:0/2249605654 >> v2:172.21.15.135:6800/116426 conn(0x7efc54068890 0x7efc5406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.699+0000 7efc677fe700 10 monclient: get_auth_request con 0x7efc54068890 auth_method 0 2023-09-30T00:51:00.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.700+0000 7efc677fe700 1 --2- 172.21.15.40:0/2249605654 >> v2:172.21.15.135:6800/116426 conn(0x7efc54068890 0x7efc5406ad40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efc58009880 tx=0x7efc58018000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:00.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.700+0000 7efc4f7fe700 10 client.4663.objecter ms_handle_connect 0x7efc54068890 2023-09-30T00:51:00.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.702+0000 7efc4f7fe700 1 -- 172.21.15.40:0/2249605654 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7efc5c0562f0 con 0x7efc680af410 2023-09-30T00:51:00.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.702+0000 7efc4f7fe700 10 client.4663.objecter ms_dispatch 0x7efc68001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:00.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.702+0000 7efc4f7fe700 3 client.4663.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:00.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.702+0000 7efc4f7fe700 3 client.4663.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:00.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.703+0000 7efc4f7fe700 20 client.4663.objecter dump_active .. 0 homeless 2023-09-30T00:51:00.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.949+0000 7efc4f7fe700 1 -- 172.21.15.40:0/2249605654 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efc5c051f30 con 0x7efc680af410 2023-09-30T00:51:00.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.949+0000 7efc4f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:00.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:00.949+0000 7efc4f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.200+0000 7efc6dad8700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.201+0000 7efc6dad8700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.201+0000 7efc6dad8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:01.201 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.201+0000 7efc6dad8700 1 -- 172.21.15.40:0/2249605654 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efc680a9fb0 con 0x7efc680af410 2023-09-30T00:51:01.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.205+0000 7efc4f7fe700 1 -- 172.21.15.40:0/2249605654 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7efc5c017100 con 0x7efc680af410 2023-09-30T00:51:01.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.206+0000 7efc4f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:01.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.206+0000 7efc4f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:01.205 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:01.206 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:01.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 10 librados: watch_flush enter 2023-09-30T00:51:01.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 10 librados: watch_flush exit 2023-09-30T00:51:01.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 20 client.4663.objecter shutdown clearing up homeless session... 2023-09-30T00:51:01.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 10 client.4663.objecter successfully canceled tick 2023-09-30T00:51:01.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 1 -- 172.21.15.40:0/2249605654 >> v2:172.21.15.135:6800/116426 conn(0x7efc54068890 msgr2=0x7efc5406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:01.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 1 --2- 172.21.15.40:0/2249605654 >> v2:172.21.15.135:6800/116426 conn(0x7efc54068890 0x7efc5406ad40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efc58009880 tx=0x7efc58018000 comp rx=0 tx=0).stop 2023-09-30T00:51:01.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 10 monclient: shutdown 2023-09-30T00:51:01.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:01.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 1 -- 172.21.15.40:0/2249605654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc680af410 msgr2=0x7efc680ae0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:01.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 1 --2- 172.21.15.40:0/2249605654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc680af410 0x7efc680ae0d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efc5c00ea50 tx=0x7efc5c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:51:01.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 1 -- 172.21.15.40:0/2249605654 shutdown_connections 2023-09-30T00:51:01.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 1 --2- 172.21.15.40:0/2249605654 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efc680b1460 0x7efc680a79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:01.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 1 --2- 172.21.15.40:0/2249605654 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc680a5fb0 0x7efc6813dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:01.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 1 --2- 172.21.15.40:0/2249605654 >> v2:172.21.15.135:6800/116426 conn(0x7efc54068890 0x7efc5406ad40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:01.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 1 --2- 172.21.15.40:0/2249605654 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efc680af410 0x7efc680ae0d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:01.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 1 -- 172.21.15.40:0/2249605654 >> 172.21.15.40:0/2249605654 conn(0x7efc680631f0 msgr2=0x7efc680b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:01.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 1 -- 172.21.15.40:0/2249605654 shutdown_connections 2023-09-30T00:51:01.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 1 -- 172.21.15.40:0/2249605654 wait complete. 2023-09-30T00:51:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:01.208+0000 7efc6dad8700 1 librados: shutdown 2023-09-30T00:51:04.222 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.364+0000 7fbabbab1700 10 monclient: get_monmap_and_config 2023-09-30T00:51:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.364+0000 7fbabbab1700 10 monclient: build_initial_monmap 2023-09-30T00:51:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.364+0000 7fbabbab1700 10 monclient: monmap: 2023-09-30T00:51:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:04.365323+0000 2023-09-30T00:51:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:04.365323+0000 2023-09-30T00:51:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:04.366 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.364+0000 7fbabbab1700 1 Processor -- start 2023-09-30T00:51:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.365+0000 7fbabbab1700 1 -- start start 2023-09-30T00:51:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.365+0000 7fbabbab1700 10 monclient: init 2023-09-30T00:51:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.365+0000 7fbabbab1700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.365+0000 7fbabbab1700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-09-30T00:51:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.365+0000 7fbabbab1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbab40b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:51:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.365+0000 7fbabbab1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbab40b1830 addr v1:172.21.15.135:6789/0 2023-09-30T00:51:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.365+0000 7fbabbab1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbab40a5fb0 0x7fbab413b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.365+0000 7fbabbab1700 10 monclient(hunting): picked mon.noname-a con 0x7fbab40a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:51:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.365+0000 7fbabbab1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab40a5b20 con 0x7fbab40b1460 2023-09-30T00:51:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.365+0000 7fbabbab1700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab40a71f0 con 0x7fbab40b1830 2023-09-30T00:51:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.365+0000 7fbabbab1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.365+0000 7fbabbab1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab4141e40 con 0x7fbab40a5fb0 2023-09-30T00:51:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.365+0000 7fbabbab1700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.365+0000 7fbabbab1700 10 monclient(hunting): authenticate will time out at 5525.666910s 2023-09-30T00:51:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.366+0000 7fbaba04e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbab40a5fb0 0x7fbab413b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.367+0000 7fbaba04e700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbab40a5fb0 0x7fbab413b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45406/0 (socket says 172.21.15.40:45406) 2023-09-30T00:51:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.367+0000 7fbaba04e700 1 -- 172.21.15.40:0/3851221001 learned_addr learned my addr 172.21.15.40:0/3851221001 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.367+0000 7fbaba04e700 10 monclient(hunting): get_auth_request con 0x7fbab40a5fb0 auth_method 0 2023-09-30T00:51:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.367+0000 7fbaba04e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.367+0000 7fbaba04e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:04.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.367+0000 7fbaba04e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:04.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.367+0000 7fbaba04e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:04.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.367+0000 7fbaba04e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:04.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.367+0000 7fbaba04e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:04.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.370+0000 7fbaba04e700 10 monclient(hunting): handle_auth_done global_id 4608 payload 274 2023-09-30T00:51:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.370+0000 7fbaba04e700 1 -- 172.21.15.40:0/3851221001 >> v1:172.21.15.135:6789/0 conn(0x7fbab40b1830 legacy=0x7fbab413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.370+0000 7fbaba04e700 1 -- 172.21.15.40:0/3851221001 >> v1:172.21.15.40:6789/0 conn(0x7fbab40b1460 legacy=0x7fbab4141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.370+0000 7fbaba04e700 10 monclient: _finish_hunting 0 2023-09-30T00:51:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.370+0000 7fbaba04e700 1 monclient: found mon.noname-a 2023-09-30T00:51:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.370+0000 7fbaba04e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.370+0000 7fbaba04e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:51:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.370+0000 7fbaba04e700 1 -- 172.21.15.40:0/3851221001 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab40a74d0 con 0x7fbab40a5fb0 2023-09-30T00:51:04.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.372+0000 7fbaba04e700 1 --2- 172.21.15.40:0/3851221001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbab40a5fb0 0x7fbab413b330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009f50 tx=0x7fbaac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3efbeef4201af415 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:04.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.391+0000 7fbab884b700 1 -- 172.21.15.40:0/3851221001 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac00e070 con 0x7fbab40a5fb0 2023-09-30T00:51:04.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.391+0000 7fbab884b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:04.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbab884b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:51:04.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbab884b700 10 monclient: dump: 2023-09-30T00:51:04.394 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:04.394 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:04.394 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:04.394 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:04.394 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:04.394 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:04.394 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:04.394 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:04.394 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:04.394 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:04.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbab884b700 10 monclient: _finish_auth 0 2023-09-30T00:51:04.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbab884b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:04.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbabbab1700 5 monclient: authenticate success, global_id 4608 2023-09-30T00:51:04.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbabbab1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:51:04.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbab884b700 1 -- 172.21.15.40:0/3851221001 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac00ba30 con 0x7fbab40a5fb0 2023-09-30T00:51:04.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbab884b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:04.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbabbab1700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:04.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbabbab1700 4 set_mon_vals no callback set 2023-09-30T00:51:04.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbab884b700 1 -- 172.21.15.40:0/3851221001 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac0092e0 con 0x7fbab40a5fb0 2023-09-30T00:51:04.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbab884b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:04.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbab884b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:51:04.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbab884b700 10 monclient: dump: 2023-09-30T00:51:04.395 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:04.396 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:04.397 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbabbab1700 10 monclient: shutdown 2023-09-30T00:51:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbabbab1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbabbab1700 1 -- 172.21.15.40:0/3851221001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbab40a5fb0 msgr2=0x7fbab413b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbabbab1700 1 --2- 172.21.15.40:0/3851221001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbab40a5fb0 0x7fbab413b330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbaac009f50 tx=0x7fbaac009840 comp rx=0 tx=0).stop 2023-09-30T00:51:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbabbab1700 1 -- 172.21.15.40:0/3851221001 shutdown_connections 2023-09-30T00:51:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbabbab1700 1 --2- 172.21.15.40:0/3851221001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbab40a5fb0 0x7fbab413b330 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbabbab1700 1 -- 172.21.15.40:0/3851221001 >> 172.21.15.40:0/3851221001 conn(0x7fbab40631f0 msgr2=0x7fbab40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.392+0000 7fbabbab1700 1 -- 172.21.15.40:0/3851221001 shutdown_connections 2023-09-30T00:51:04.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.393+0000 7fbabbab1700 1 -- 172.21.15.40:0/3851221001 wait complete. 2023-09-30T00:51:04.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.393+0000 7fbabbab1700 10 monclient: build_initial_monmap 2023-09-30T00:51:04.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.393+0000 7fbabbab1700 10 monclient: monmap: 2023-09-30T00:51:04.400 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:04.400 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:04.400 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:04.400 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:04.400 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:04.400 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:04.400 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:04.400 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:04.400 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:04.401 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:04.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.393+0000 7fbabbab1700 1 librados: starting msgr at 2023-09-30T00:51:04.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.393+0000 7fbabbab1700 1 librados: starting objecter 2023-09-30T00:51:04.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 1 Processor -- start 2023-09-30T00:51:04.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 1 -- start start 2023-09-30T00:51:04.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 1 librados: setting wanted keys 2023-09-30T00:51:04.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 1 librados: calling monclient init 2023-09-30T00:51:04.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 10 monclient: init 2023-09-30T00:51:04.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:04.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:51:04.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbab40a5fb0 0x7fbab40a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:04.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 10 monclient(hunting): picked mon.noname-b con 0x7fbab40a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:04.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab40b1460 0x7fbab413e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:04.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 10 monclient(hunting): picked mon.noname-c con 0x7fbab40b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:04.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbab40b1830 0x7fbab40aa050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:04.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 10 monclient(hunting): picked mon.noname-a con 0x7fbab40b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:04.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:04.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40aa7b0 con 0x7fbab40b1830 2023-09-30T00:51:04.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:04.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40a79f0 con 0x7fbab40a5fb0 2023-09-30T00:51:04.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:04.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab40a7b60 con 0x7fbab40b1460 2023-09-30T00:51:04.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:04.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.394+0000 7fbabbab1700 10 monclient(hunting): authenticate will time out at 5525.695678s 2023-09-30T00:51:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.395+0000 7fbab904c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab40b1460 0x7fbab413e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.395+0000 7fbab984d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbab40a5fb0 0x7fbab40a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.395+0000 7fbab904c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab40b1460 0x7fbab413e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:52168/0 (socket says 172.21.15.40:52168) 2023-09-30T00:51:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.395+0000 7fbab904c700 1 -- 172.21.15.40:0/2754066388 learned_addr learned my addr 172.21.15.40:0/2754066388 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.395+0000 7fbab984d700 10 monclient(hunting): get_auth_request con 0x7fbab40a5fb0 auth_method 0 2023-09-30T00:51:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.395+0000 7fbab984d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.395+0000 7fbab984d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.395+0000 7fbab984d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.395+0000 7fbab904c700 10 monclient(hunting): get_auth_request con 0x7fbab40b1460 auth_method 0 2023-09-30T00:51:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.395+0000 7fbab904c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.395+0000 7fbab904c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.395+0000 7fbab904c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:04.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.396+0000 7fbab904c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:04.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.396+0000 7fbab904c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:04.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.396+0000 7fbab904c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:04.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.396+0000 7fbab984d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:04.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.396+0000 7fbab984d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:04.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.396+0000 7fbab984d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:04.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.399+0000 7fbaba04e700 1 --2- 172.21.15.40:0/2754066388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbab40b1830 0x7fbab40aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.400+0000 7fbaba04e700 10 monclient(hunting): get_auth_request con 0x7fbab40b1830 auth_method 0 2023-09-30T00:51:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.400+0000 7fbaba04e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.400+0000 7fbaba04e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.400+0000 7fbaba04e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:04.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.400+0000 7fbaba04e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:04.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.400+0000 7fbaba04e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:04.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.400+0000 7fbaba04e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:04.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.401+0000 7fbab984d700 10 monclient(hunting): handle_auth_done global_id 4666 payload 824 2023-09-30T00:51:04.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.401+0000 7fbab984d700 1 -- 172.21.15.40:0/2754066388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab40b1460 msgr2=0x7fbab413e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:04.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.401+0000 7fbab984d700 1 --2- 172.21.15.40:0/2754066388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab40b1460 0x7fbab413e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:04.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.401+0000 7fbab984d700 1 -- 172.21.15.40:0/2754066388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbab40b1830 msgr2=0x7fbab40aa050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:04.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.401+0000 7fbab984d700 1 --2- 172.21.15.40:0/2754066388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbab40b1830 0x7fbab40aa050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:04.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.401+0000 7fbab984d700 10 monclient: _finish_hunting 0 2023-09-30T00:51:04.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.401+0000 7fbab984d700 1 monclient: found mon.noname-b 2023-09-30T00:51:04.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.401+0000 7fbab984d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:04.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.401+0000 7fbab984d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:51:04.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.401+0000 7fbab984d700 1 -- 172.21.15.40:0/2754066388 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaac009ae0 con 0x7fbab40a5fb0 2023-09-30T00:51:04.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.401+0000 7fbab904c700 1 --2- 172.21.15.40:0/2754066388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab40b1460 0x7fbab413e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:51:04.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.404+0000 7fbab984d700 1 --2- 172.21.15.40:0/2754066388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbab40a5fb0 0x7fbab40a9a70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbab000b5b0 tx=0x7fbab00090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:04.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.404+0000 7fbaa5ffb700 10 client.?.objecter ms_handle_connect 0x7fbab40a5fb0 2023-09-30T00:51:04.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.404+0000 7fbaa5ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:04.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbaa5ffb700 1 -- 172.21.15.40:0/2754066388 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab000f510 con 0x7fbab40a5fb0 2023-09-30T00:51:04.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbaa5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:04.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbaa5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:04.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbaa5ffb700 10 monclient: dump: 2023-09-30T00:51:04.411 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:04.413 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbaa5ffb700 10 monclient: _finish_auth 0 2023-09-30T00:51:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbaa5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbaa5ffb700 1 -- 172.21.15.40:0/2754066388 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab000f690 con 0x7fbab40a5fb0 2023-09-30T00:51:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbaa5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbaa5ffb700 1 -- 172.21.15.40:0/2754066388 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab000fa50 con 0x7fbab40a5fb0 2023-09-30T00:51:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbaa5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbaa5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbaa5ffb700 10 monclient: dump: 2023-09-30T00:51:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:04.414 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbabbab1700 5 monclient: authenticate success, global_id 4666 2023-09-30T00:51:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbabbab1700 10 monclient: _renew_subs 2023-09-30T00:51:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbabbab1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbabbab1700 1 -- 172.21.15.40:0/2754066388 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab40a7de0 con 0x7fbab40a5fb0 2023-09-30T00:51:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.410+0000 7fbaa7fff700 4 set_mon_vals no callback set 2023-09-30T00:51:04.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.411+0000 7fbabbab1700 10 client.4666.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:04.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.411+0000 7fbabbab1700 10 monclient: _renew_subs 2023-09-30T00:51:04.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.411+0000 7fbabbab1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:04.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.411+0000 7fbabbab1700 1 -- 172.21.15.40:0/2754066388 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab40a8300 con 0x7fbab40a5fb0 2023-09-30T00:51:04.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.411+0000 7fbabbab1700 1 librados: init done 2023-09-30T00:51:04.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.412+0000 7fbabbab1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:04.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.412+0000 7fbabbab1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:04.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.412+0000 7fbabbab1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:04.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.412+0000 7fbabbab1700 1 -- 172.21.15.40:0/2754066388 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab40aa590 con 0x7fbab40a5fb0 2023-09-30T00:51:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.660+0000 7fbaa5ffb700 1 -- 172.21.15.40:0/2754066388 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fbab0024430 con 0x7fbab40a5fb0 2023-09-30T00:51:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.661+0000 7fbaa5ffb700 1 --2- 172.21.15.40:0/2754066388 >> v2:172.21.15.135:6800/116426 conn(0x7fba98068990 0x7fba9806ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:04.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.661+0000 7fbab904c700 1 --2- 172.21.15.40:0/2754066388 >> v2:172.21.15.135:6800/116426 conn(0x7fba98068990 0x7fba9806ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:04.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.661+0000 7fbab904c700 10 monclient: get_auth_request con 0x7fba98068990 auth_method 0 2023-09-30T00:51:04.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.661+0000 7fbaa5ffb700 10 client.4666.objecter ms_handle_connect 0x7fba98068990 2023-09-30T00:51:04.662 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.661+0000 7fbab904c700 1 --2- 172.21.15.40:0/2754066388 >> v2:172.21.15.135:6800/116426 conn(0x7fba98068990 0x7fba9806ae40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbab413c070 tx=0x7fbaa80090f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.664+0000 7fbaa5ffb700 1 -- 172.21.15.40:0/2754066388 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fbab0055f70 con 0x7fbab40a5fb0 2023-09-30T00:51:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.664+0000 7fbaa5ffb700 10 client.4666.objecter ms_dispatch 0x7fbab4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.664+0000 7fbaa5ffb700 3 client.4666.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.664+0000 7fbaa5ffb700 3 client.4666.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.665+0000 7fbaa5ffb700 20 client.4666.objecter dump_active .. 0 homeless 2023-09-30T00:51:04.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.678+0000 7fbaa5ffb700 1 -- 172.21.15.40:0/2754066388 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbab0051bb0 con 0x7fbab40a5fb0 2023-09-30T00:51:04.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.678+0000 7fbaa5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:04.679 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.678+0000 7fbaa5ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:04.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.942+0000 7fbabbab1700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:04.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.942+0000 7fbabbab1700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:04.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.942+0000 7fbabbab1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:04.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.942+0000 7fbabbab1700 1 -- 172.21.15.40:0/2754066388 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbab4064a00 con 0x7fbab40a5fb0 2023-09-30T00:51:04.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.946+0000 7fbaa5ffb700 1 -- 172.21.15.40:0/2754066388 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbab00551d0 con 0x7fbab40a5fb0 2023-09-30T00:51:04.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.946+0000 7fbaa5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:04.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.946+0000 7fbaa5ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:04.947 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:04.947 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.948+0000 7fbabbab1700 10 librados: watch_flush enter 2023-09-30T00:51:04.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.948+0000 7fbabbab1700 10 librados: watch_flush exit 2023-09-30T00:51:04.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.948+0000 7fbabbab1700 20 client.4666.objecter shutdown clearing up homeless session... 2023-09-30T00:51:04.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.948+0000 7fbabbab1700 10 client.4666.objecter successfully canceled tick 2023-09-30T00:51:04.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.948+0000 7fbabbab1700 1 -- 172.21.15.40:0/2754066388 >> v2:172.21.15.135:6800/116426 conn(0x7fba98068990 msgr2=0x7fba9806ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:04.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.948+0000 7fbabbab1700 1 --2- 172.21.15.40:0/2754066388 >> v2:172.21.15.135:6800/116426 conn(0x7fba98068990 0x7fba9806ae40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbab413c070 tx=0x7fbaa80090f0 comp rx=0 tx=0).stop 2023-09-30T00:51:04.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.948+0000 7fbabbab1700 10 monclient: shutdown 2023-09-30T00:51:04.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.948+0000 7fbabbab1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:04.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.948+0000 7fbabbab1700 1 -- 172.21.15.40:0/2754066388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbab40a5fb0 msgr2=0x7fbab40a9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:04.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.948+0000 7fbabbab1700 1 --2- 172.21.15.40:0/2754066388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbab40a5fb0 0x7fbab40a9a70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbab000b5b0 tx=0x7fbab00090f0 comp rx=0 tx=0).stop 2023-09-30T00:51:04.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.949+0000 7fbabbab1700 1 -- 172.21.15.40:0/2754066388 shutdown_connections 2023-09-30T00:51:04.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.949+0000 7fbabbab1700 1 --2- 172.21.15.40:0/2754066388 >> v2:172.21.15.135:6800/116426 conn(0x7fba98068990 0x7fba9806ae40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:04.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.949+0000 7fbabbab1700 1 --2- 172.21.15.40:0/2754066388 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbab40a5fb0 0x7fbab40a9a70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:04.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.949+0000 7fbabbab1700 1 --2- 172.21.15.40:0/2754066388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbab40b1830 0x7fbab40aa050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:04.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.949+0000 7fbabbab1700 1 --2- 172.21.15.40:0/2754066388 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbab40b1460 0x7fbab413e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:04.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.949+0000 7fbabbab1700 1 -- 172.21.15.40:0/2754066388 >> 172.21.15.40:0/2754066388 conn(0x7fbab40631f0 msgr2=0x7fbab413f910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:04.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.949+0000 7fbabbab1700 1 -- 172.21.15.40:0/2754066388 shutdown_connections 2023-09-30T00:51:04.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.949+0000 7fbabbab1700 1 -- 172.21.15.40:0/2754066388 wait complete. 2023-09-30T00:51:04.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:04.949+0000 7fbabbab1700 1 librados: shutdown 2023-09-30T00:51:07.964 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:08.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.107+0000 7f91a7165700 10 monclient: get_monmap_and_config 2023-09-30T00:51:08.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.107+0000 7f91a7165700 10 monclient: build_initial_monmap 2023-09-30T00:51:08.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.107+0000 7f91a7165700 10 monclient: monmap: 2023-09-30T00:51:08.108 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:08.108 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:08.109 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:08.108854+0000 2023-09-30T00:51:08.109 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:08.108854+0000 2023-09-30T00:51:08.109 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:08.109 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:08.109 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:08.109 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:08.109 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:08.109 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:08.109 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:08.109 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:08.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.108+0000 7f91a7165700 1 Processor -- start 2023-09-30T00:51:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.108+0000 7f91a7165700 1 -- start start 2023-09-30T00:51:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.108+0000 7f91a7165700 10 monclient: init 2023-09-30T00:51:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.109+0000 7f91a7165700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.109+0000 7f91a7165700 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-09-30T00:51:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.109+0000 7f91a7165700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91a00a5710 addr v1:172.21.15.135:6789/0 2023-09-30T00:51:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.109+0000 7f91a7165700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91a013e1d0 0x7f91a013e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.109+0000 7f91a7165700 10 monclient(hunting): picked mon.noname-b con 0x7f91a013e1d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.109+0000 7f91a7165700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f91a013eb60 0x7f91a013ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.109+0000 7f91a7165700 10 monclient(hunting): picked mon.noname-c con 0x7f91a013eb60 addr v2:172.21.15.135:3300/0 2023-09-30T00:51:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.109+0000 7f91a7165700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a00a5ae0 con 0x7f91a00a5710 2023-09-30T00:51:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.109+0000 7f91a7165700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.109+0000 7f91a7165700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a00a7ff0 con 0x7f91a013eb60 2023-09-30T00:51:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.109+0000 7f91a7165700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.109+0000 7f91a7165700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a00a7cf0 con 0x7f91a013e1d0 2023-09-30T00:51:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.109+0000 7f91a7165700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.109+0000 7f91a7165700 10 monclient(hunting): authenticate will time out at 5529.410428s 2023-09-30T00:51:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.115+0000 7f91a5702700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f91a013eb60 0x7f91a013ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.116+0000 7f91a5702700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f91a013eb60 0x7f91a013ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:53726/0 (socket says 172.21.15.40:53726) 2023-09-30T00:51:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.116+0000 7f91a5702700 1 -- 172.21.15.40:0/1929864447 learned_addr learned my addr 172.21.15.40:0/1929864447 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.116+0000 7f91a5702700 10 monclient(hunting): get_auth_request con 0x7f91a013eb60 auth_method 0 2023-09-30T00:51:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.116+0000 7f91a5702700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.116+0000 7f91a5702700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.116+0000 7f91a5702700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.116+0000 7f91a5702700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.116+0000 7f91a5702700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:08.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.116+0000 7f91a5702700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.119+0000 7f91a5702700 10 monclient(hunting): handle_auth_done global_id 4669 payload 274 2023-09-30T00:51:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.119+0000 7f91a5702700 1 -- 172.21.15.40:0/1929864447 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91a013e1d0 msgr2=0x7f91a013e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.119+0000 7f91a5702700 1 --2- 172.21.15.40:0/1929864447 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91a013e1d0 0x7f91a013e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.119+0000 7f91a5702700 1 -- 172.21.15.40:0/1929864447 >> v1:172.21.15.135:6789/0 conn(0x7f91a00a5710 legacy=0x7f91a00a75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.119+0000 7f91a5702700 10 monclient: _finish_hunting 0 2023-09-30T00:51:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.119+0000 7f91a5702700 1 monclient: found mon.noname-c 2023-09-30T00:51:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.119+0000 7f91a5702700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.119+0000 7f91a5702700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:51:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.119+0000 7f91a5702700 1 -- 172.21.15.40:0/1929864447 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a00aedc0 con 0x7f91a013eb60 2023-09-30T00:51:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.121+0000 7f91a5702700 1 --2- 172.21.15.40:0/1929864447 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91a013eb60 0x7f91a013ae40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f91900099b0 tx=0x7f9190009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38b079bd314fe6ba server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:08.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f919f7fe700 1 -- 172.21.15.40:0/1929864447 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919000b8b0 con 0x7f91a013eb60 2023-09-30T00:51:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f919f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f919f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:51:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f919f7fe700 10 monclient: dump: 2023-09-30T00:51:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:08.128 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f919f7fe700 10 monclient: _finish_auth 0 2023-09-30T00:51:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f919f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f919f7fe700 1 -- 172.21.15.40:0/1929864447 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919000ba10 con 0x7f91a013eb60 2023-09-30T00:51:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f91a7165700 5 monclient: authenticate success, global_id 4669 2023-09-30T00:51:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f91a7165700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:51:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f919f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f91a7165700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f91a7165700 4 set_mon_vals no callback set 2023-09-30T00:51:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f919f7fe700 1 -- 172.21.15.40:0/1929864447 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919000bdb0 con 0x7f91a013eb60 2023-09-30T00:51:08.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f919f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:08.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f919f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:08.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.126+0000 7f919f7fe700 10 monclient: dump: 2023-09-30T00:51:08.129 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:08.129 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:08.129 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:08.129 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:08.129 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:08.129 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:08.130 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:08.130 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:08.130 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:08.130 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.127+0000 7f91a7165700 10 monclient: shutdown 2023-09-30T00:51:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.127+0000 7f91a7165700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.127+0000 7f91a7165700 1 -- 172.21.15.40:0/1929864447 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91a013eb60 msgr2=0x7f91a013ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.127+0000 7f91a7165700 1 --2- 172.21.15.40:0/1929864447 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91a013eb60 0x7f91a013ae40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f91900099b0 tx=0x7f9190009250 comp rx=0 tx=0).stop 2023-09-30T00:51:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.127+0000 7f91a7165700 1 -- 172.21.15.40:0/1929864447 shutdown_connections 2023-09-30T00:51:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.127+0000 7f91a7165700 1 --2- 172.21.15.40:0/1929864447 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91a013eb60 0x7f91a013ae40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.127+0000 7f91a7165700 1 --2- 172.21.15.40:0/1929864447 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91a013e1d0 0x7f91a013e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.127+0000 7f91a7165700 1 -- 172.21.15.40:0/1929864447 >> 172.21.15.40:0/1929864447 conn(0x7f91a0069080 msgr2=0x7f91a00631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.127+0000 7f91a7165700 1 -- 172.21.15.40:0/1929864447 shutdown_connections 2023-09-30T00:51:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.127+0000 7f91a7165700 1 -- 172.21.15.40:0/1929864447 wait complete. 2023-09-30T00:51:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.127+0000 7f91a7165700 10 monclient: build_initial_monmap 2023-09-30T00:51:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.127+0000 7f91a7165700 10 monclient: monmap: 2023-09-30T00:51:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:08.135 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.128+0000 7f91a7165700 1 librados: starting msgr at 2023-09-30T00:51:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.128+0000 7f91a7165700 1 librados: starting objecter 2023-09-30T00:51:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.128+0000 7f91a7165700 1 Processor -- start 2023-09-30T00:51:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.128+0000 7f91a7165700 1 -- start start 2023-09-30T00:51:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.128+0000 7f91a7165700 1 librados: setting wanted keys 2023-09-30T00:51:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.128+0000 7f91a7165700 1 librados: calling monclient init 2023-09-30T00:51:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.128+0000 7f91a7165700 10 monclient: init 2023-09-30T00:51:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:51:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91a00a5710 0x7f91a0151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 10 monclient(hunting): picked mon.noname-b con 0x7f91a00a5710 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91a013e1d0 0x7f91a00b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 10 monclient(hunting): picked mon.noname-a con 0x7f91a013e1d0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91a013eb60 0x7f91a00a96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 10 monclient(hunting): picked mon.noname-c con 0x7f91a013eb60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a0151a90 con 0x7f91a013e1d0 2023-09-30T00:51:08.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:08.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a00b28d0 con 0x7f91a00a5710 2023-09-30T00:51:08.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:08.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a0151f20 con 0x7f91a013eb60 2023-09-30T00:51:08.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:08.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a7165700 10 monclient(hunting): authenticate will time out at 5529.430229s 2023-09-30T00:51:08.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a4f01700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91a00a5710 0x7f91a0151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:08.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f91a5702700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91a013eb60 0x7f91a00a96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:08.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.129+0000 7f919ffff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91a013e1d0 0x7f91a00b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:08.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f91a4f01700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91a00a5710 0x7f91a0151340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:53730/0 (socket says 172.21.15.40:53730) 2023-09-30T00:51:08.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f91a4f01700 1 -- 172.21.15.40:0/2917703 learned_addr learned my addr 172.21.15.40:0/2917703 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:08.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f91a4f01700 10 monclient(hunting): get_auth_request con 0x7f91a00a5710 auth_method 0 2023-09-30T00:51:08.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f91a4f01700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:08.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f91a4f01700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:08.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f91a4f01700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:08.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f91a5702700 10 monclient(hunting): get_auth_request con 0x7f91a013eb60 auth_method 0 2023-09-30T00:51:08.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f91a5702700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:08.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f91a5702700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:08.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f91a5702700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:08.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f919ffff700 10 monclient(hunting): get_auth_request con 0x7f91a013e1d0 auth_method 0 2023-09-30T00:51:08.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f919ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:08.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f919ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:08.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f919ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:08.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f91a4f01700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:08.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f91a4f01700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:08.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.130+0000 7f91a4f01700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:08.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.131+0000 7f91a5702700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:08.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.131+0000 7f91a5702700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:08.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.131+0000 7f91a5702700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:08.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.131+0000 7f919ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:08.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.131+0000 7f919ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:08.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.131+0000 7f919ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:08.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.133+0000 7f91a4f01700 10 monclient(hunting): handle_auth_done global_id 4672 payload 824 2023-09-30T00:51:08.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.134+0000 7f91a4f01700 1 -- 172.21.15.40:0/2917703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91a013eb60 msgr2=0x7f91a00a96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:08.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.134+0000 7f91a4f01700 1 --2- 172.21.15.40:0/2917703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91a013eb60 0x7f91a00a96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:08.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.134+0000 7f91a4f01700 1 -- 172.21.15.40:0/2917703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91a013e1d0 msgr2=0x7f91a00b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:08.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.134+0000 7f91a4f01700 1 --2- 172.21.15.40:0/2917703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91a013e1d0 0x7f91a00b3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:08.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.134+0000 7f91a4f01700 10 monclient: _finish_hunting 0 2023-09-30T00:51:08.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.134+0000 7f91a4f01700 1 monclient: found mon.noname-b 2023-09-30T00:51:08.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.134+0000 7f91a4f01700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:08.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.134+0000 7f91a4f01700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:51:08.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.134+0000 7f91a4f01700 1 -- 172.21.15.40:0/2917703 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91900094f0 con 0x7f91a00a5710 2023-09-30T00:51:08.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.136+0000 7f91a4f01700 1 --2- 172.21.15.40:0/2917703 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91a00a5710 0x7f91a0151340 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f919800bbe0 tx=0x7f9198009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:08.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.136+0000 7f919cff9700 10 client.?.objecter ms_handle_connect 0x7f91a00a5710 2023-09-30T00:51:08.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.136+0000 7f919cff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:08.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f919cff9700 1 -- 172.21.15.40:0/2917703 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91980092e0 con 0x7f91a00a5710 2023-09-30T00:51:08.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f919cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:08.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f919cff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:08.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f919cff9700 10 monclient: dump: 2023-09-30T00:51:08.143 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:08.143 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:08.143 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:08.143 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:08.143 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:08.143 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:08.143 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:08.143 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:08.143 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:08.143 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:08.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f919cff9700 10 monclient: _finish_auth 0 2023-09-30T00:51:08.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f919cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:08.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f919cff9700 1 -- 172.21.15.40:0/2917703 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919800c630 con 0x7f91a00a5710 2023-09-30T00:51:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f91a7165700 5 monclient: authenticate success, global_id 4672 2023-09-30T00:51:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f919cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f919cff9700 1 -- 172.21.15.40:0/2917703 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919800c9d0 con 0x7f91a00a5710 2023-09-30T00:51:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f91a7165700 10 monclient: _renew_subs 2023-09-30T00:51:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f91a7165700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f91a7165700 1 -- 172.21.15.40:0/2917703 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a00b57e0 con 0x7f91a00a5710 2023-09-30T00:51:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f919effd700 4 set_mon_vals no callback set 2023-09-30T00:51:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f919cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f919cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f919cff9700 10 monclient: dump: 2023-09-30T00:51:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:08.145 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f91a7165700 10 client.4672.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f91a7165700 10 monclient: _renew_subs 2023-09-30T00:51:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f91a7165700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f91a7165700 1 -- 172.21.15.40:0/2917703 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a00a9e60 con 0x7f91a00a5710 2023-09-30T00:51:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.140+0000 7f91a7165700 1 librados: init done 2023-09-30T00:51:08.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.141+0000 7f91a7165700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:08.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.141+0000 7f91a7165700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:08.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.141+0000 7f91a7165700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:08.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.141+0000 7f91a7165700 1 -- 172.21.15.40:0/2917703 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a0072b90 con 0x7f91a00a5710 2023-09-30T00:51:08.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.160+0000 7f919cff9700 1 -- 172.21.15.40:0/2917703 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9198012070 con 0x7f91a00a5710 2023-09-30T00:51:08.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.161+0000 7f919cff9700 1 --2- 172.21.15.40:0/2917703 >> v2:172.21.15.135:6800/116426 conn(0x7f9180068960 0x7f918006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.161+0000 7f919ffff700 1 --2- 172.21.15.40:0/2917703 >> v2:172.21.15.135:6800/116426 conn(0x7f9180068960 0x7f918006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.161+0000 7f919ffff700 10 monclient: get_auth_request con 0x7f9180068960 auth_method 0 2023-09-30T00:51:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.162+0000 7f919ffff700 1 --2- 172.21.15.40:0/2917703 >> v2:172.21.15.135:6800/116426 conn(0x7f9180068960 0x7f918006ae10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f91a00aea90 tx=0x7f91880091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.162+0000 7f919cff9700 10 client.4672.objecter ms_handle_connect 0x7f9180068960 2023-09-30T00:51:08.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.356+0000 7f919cff9700 1 -- 172.21.15.40:0/2917703 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9198055a90 con 0x7f91a00a5710 2023-09-30T00:51:08.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.356+0000 7f919cff9700 10 client.4672.objecter ms_dispatch 0x7f91a0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:08.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.356+0000 7f919cff9700 3 client.4672.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:08.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.356+0000 7f919cff9700 3 client.4672.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:08.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.357+0000 7f919cff9700 20 client.4672.objecter dump_active .. 0 homeless 2023-09-30T00:51:08.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.426+0000 7f919cff9700 1 -- 172.21.15.40:0/2917703 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9198051500 con 0x7f91a00a5710 2023-09-30T00:51:08.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.426+0000 7f919cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:08.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.426+0000 7f919cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:08.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.700+0000 7f91a7165700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:08.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.700+0000 7f91a7165700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:08.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.700+0000 7f91a7165700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:08.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.700+0000 7f91a7165700 1 -- 172.21.15.40:0/2917703 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f91a00b1b00 con 0x7f91a00a5710 2023-09-30T00:51:08.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.704+0000 7f919cff9700 1 -- 172.21.15.40:0/2917703 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9198054cf0 con 0x7f91a00a5710 2023-09-30T00:51:08.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.704+0000 7f919cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:08.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.704+0000 7f919cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:08.705 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:08.705 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.706+0000 7f91a7165700 10 librados: watch_flush enter 2023-09-30T00:51:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 10 librados: watch_flush exit 2023-09-30T00:51:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 20 client.4672.objecter shutdown clearing up homeless session... 2023-09-30T00:51:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 10 client.4672.objecter successfully canceled tick 2023-09-30T00:51:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 1 -- 172.21.15.40:0/2917703 >> v2:172.21.15.135:6800/116426 conn(0x7f9180068960 msgr2=0x7f918006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 1 --2- 172.21.15.40:0/2917703 >> v2:172.21.15.135:6800/116426 conn(0x7f9180068960 0x7f918006ae10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f91a00aea90 tx=0x7f91880091a0 comp rx=0 tx=0).stop 2023-09-30T00:51:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 10 monclient: shutdown 2023-09-30T00:51:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 1 -- 172.21.15.40:0/2917703 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91a00a5710 msgr2=0x7f91a0151340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:08.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 1 --2- 172.21.15.40:0/2917703 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91a00a5710 0x7f91a0151340 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f919800bbe0 tx=0x7f9198009840 comp rx=0 tx=0).stop 2023-09-30T00:51:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 1 -- 172.21.15.40:0/2917703 shutdown_connections 2023-09-30T00:51:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 1 --2- 172.21.15.40:0/2917703 >> v2:172.21.15.135:6800/116426 conn(0x7f9180068960 0x7f918006ae10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 1 --2- 172.21.15.40:0/2917703 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f91a00a5710 0x7f91a0151340 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 1 --2- 172.21.15.40:0/2917703 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f91a013eb60 0x7f91a00a96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 1 --2- 172.21.15.40:0/2917703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91a013e1d0 0x7f91a00b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 1 -- 172.21.15.40:0/2917703 >> 172.21.15.40:0/2917703 conn(0x7f91a0069080 msgr2=0x7f91a00afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 1 -- 172.21.15.40:0/2917703 shutdown_connections 2023-09-30T00:51:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 1 -- 172.21.15.40:0/2917703 wait complete. 2023-09-30T00:51:08.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:08.707+0000 7f91a7165700 1 librados: shutdown 2023-09-30T00:51:11.722 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:11.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.866+0000 7f511d001700 10 monclient: get_monmap_and_config 2023-09-30T00:51:11.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.866+0000 7f511d001700 10 monclient: build_initial_monmap 2023-09-30T00:51:11.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.866+0000 7f511d001700 10 monclient: monmap: 2023-09-30T00:51:11.867 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:11.867 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:11.867 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:11.867561+0000 2023-09-30T00:51:11.867 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:11.867561+0000 2023-09-30T00:51:11.867 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:11.867 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:11.867 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:11.868 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:11.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.867+0000 7f511d001700 1 Processor -- start 2023-09-30T00:51:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.867+0000 7f511d001700 1 -- start start 2023-09-30T00:51:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.867+0000 7f511d001700 10 monclient: init 2023-09-30T00:51:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511d001700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511d001700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-09-30T00:51:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511d001700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f51180b1460 0x7f51180b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511d001700 10 monclient(hunting): picked mon.noname-c con 0x7f51180b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:51:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511d001700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f51180a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:51:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511d001700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f511813ac20 0x7f511813b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511d001700 10 monclient(hunting): picked mon.noname-b con 0x7f511813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511d001700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51180a5b20 con 0x7f51180a5fb0 2023-09-30T00:51:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511d001700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511d001700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51180a71f0 con 0x7f51180b1460 2023-09-30T00:51:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511d001700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511d001700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51180a74d0 con 0x7f511813ac20 2023-09-30T00:51:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511d001700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511d001700 10 monclient(hunting): authenticate will time out at 5533.169145s 2023-09-30T00:51:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.868+0000 7f511759e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f511813ac20 0x7f511813b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.869+0000 7f511659c700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f51180a5fb0 0x7f5118141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:39936/0 (socket says 172.21.15.40:39936) 2023-09-30T00:51:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.869+0000 7f511659c700 1 -- 172.21.15.40:0/511686209 learned_addr learned my addr 172.21.15.40:0/511686209 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:51:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.869+0000 7f511759e700 10 monclient(hunting): get_auth_request con 0x7f511813ac20 auth_method 0 2023-09-30T00:51:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.869+0000 7f511759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.869+0000 7f511759e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:11.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.869+0000 7f511759e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:11.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.869+0000 7f5116d9d700 1 --2- 172.21.15.40:0/511686209 >> v2:172.21.15.135:3300/0 conn(0x7f51180b1460 0x7f51180b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.870+0000 7f511759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.870+0000 7f511759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.870+0000 7f511759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.870+0000 7f5116d9d700 10 monclient(hunting): get_auth_request con 0x7f51180b1460 auth_method 0 2023-09-30T00:51:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.870+0000 7f5116d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.870+0000 7f5116d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.870+0000 7f5116d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.871+0000 7f5116d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.871+0000 7f5116d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:11.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.871+0000 7f5116d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.872+0000 7f511759e700 10 monclient(hunting): handle_auth_done global_id 4547 payload 274 2023-09-30T00:51:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.872+0000 7f511759e700 1 -- 172.21.15.40:0/511686209 >> v2:172.21.15.135:3300/0 conn(0x7f51180b1460 msgr2=0x7f51180b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.872+0000 7f511759e700 1 --2- 172.21.15.40:0/511686209 >> v2:172.21.15.135:3300/0 conn(0x7f51180b1460 0x7f51180b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.872+0000 7f511759e700 1 -- 172.21.15.40:0/511686209 >> v1:172.21.15.135:6789/0 conn(0x7f51180a5fb0 legacy=0x7f5118141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:11.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.872+0000 7f511759e700 10 monclient: _finish_hunting 0 2023-09-30T00:51:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.872+0000 7f511759e700 1 monclient: found mon.noname-b 2023-09-30T00:51:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.872+0000 7f511759e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.872+0000 7f511759e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:51:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.872+0000 7f511759e700 1 -- 172.21.15.40:0/511686209 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51180a63c0 con 0x7f511813ac20 2023-09-30T00:51:11.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.875+0000 7f511759e700 1 --2- 172.21.15.40:0/511686209 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f511813ac20 0x7f511813b080 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5110009e60 tx=0x7f5110009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fa1f9f847be367f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:11.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.877+0000 7f5115d9b700 1 -- 172.21.15.40:0/511686209 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511000e070 con 0x7f511813ac20 2023-09-30T00:51:11.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.877+0000 7f5115d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:11.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.877+0000 7f5115d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:11.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.877+0000 7f5115d9b700 10 monclient: dump: 2023-09-30T00:51:11.878 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:11.878 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:11.878 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:11.879 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.877+0000 7f5115d9b700 10 monclient: _finish_auth 0 2023-09-30T00:51:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.877+0000 7f5115d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.877+0000 7f511d001700 5 monclient: authenticate success, global_id 4547 2023-09-30T00:51:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.877+0000 7f511d001700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:51:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.880+0000 7f5115d9b700 1 -- 172.21.15.40:0/511686209 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f511000bb60 con 0x7f511813ac20 2023-09-30T00:51:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.880+0000 7f5115d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.880+0000 7f5115d9b700 1 -- 172.21.15.40:0/511686209 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5110012430 con 0x7f511813ac20 2023-09-30T00:51:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.880+0000 7f511d001700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.880+0000 7f511d001700 4 set_mon_vals no callback set 2023-09-30T00:51:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.880+0000 7f5115d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.880+0000 7f5115d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:51:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.880+0000 7f5115d9b700 10 monclient: dump: 2023-09-30T00:51:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:11.881 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:11.882 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:11.882 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:11.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.881+0000 7f511d001700 10 monclient: shutdown 2023-09-30T00:51:11.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.881+0000 7f511d001700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:11.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.881+0000 7f511d001700 1 -- 172.21.15.40:0/511686209 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f511813ac20 msgr2=0x7f511813b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:11.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.881+0000 7f511d001700 1 --2- 172.21.15.40:0/511686209 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f511813ac20 0x7f511813b080 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5110009e60 tx=0x7f5110009780 comp rx=0 tx=0).stop 2023-09-30T00:51:11.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.881+0000 7f511d001700 1 -- 172.21.15.40:0/511686209 shutdown_connections 2023-09-30T00:51:11.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.881+0000 7f511d001700 1 --2- 172.21.15.40:0/511686209 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f511813ac20 0x7f511813b080 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:11.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.881+0000 7f511d001700 1 --2- 172.21.15.40:0/511686209 >> v2:172.21.15.135:3300/0 conn(0x7f51180b1460 0x7f51180b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:11.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.881+0000 7f511d001700 1 -- 172.21.15.40:0/511686209 >> 172.21.15.40:0/511686209 conn(0x7f51180631f0 msgr2=0x7f51180635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:11.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.881+0000 7f511d001700 1 -- 172.21.15.40:0/511686209 shutdown_connections 2023-09-30T00:51:11.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.881+0000 7f511d001700 1 -- 172.21.15.40:0/511686209 wait complete. 2023-09-30T00:51:11.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.882+0000 7f511d001700 10 monclient: build_initial_monmap 2023-09-30T00:51:11.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.882+0000 7f511d001700 10 monclient: monmap: 2023-09-30T00:51:11.885 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:11.885 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:11.885 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:11.885 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:11.885 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:11.885 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:11.886 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.882+0000 7f511d001700 1 librados: starting msgr at 2023-09-30T00:51:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.882+0000 7f511d001700 1 librados: starting objecter 2023-09-30T00:51:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.882+0000 7f511d001700 1 Processor -- start 2023-09-30T00:51:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.882+0000 7f511d001700 1 -- start start 2023-09-30T00:51:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.882+0000 7f511d001700 1 librados: setting wanted keys 2023-09-30T00:51:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.882+0000 7f511d001700 1 librados: calling monclient init 2023-09-30T00:51:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.882+0000 7f511d001700 10 monclient: init 2023-09-30T00:51:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:51:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51180a5fb0 0x7f511813e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 10 monclient(hunting): picked mon.noname-b con 0x7f51180a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51180b1460 0x7f511814d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 10 monclient(hunting): picked mon.noname-a con 0x7f51180b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f511813ac20 0x7f511814dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 10 monclient(hunting): picked mon.noname-c con 0x7f511813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51180a9dd0 con 0x7f51180b1460 2023-09-30T00:51:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511813eaa0 con 0x7f51180a5fb0 2023-09-30T00:51:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511813ec10 con 0x7f511813ac20 2023-09-30T00:51:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.883+0000 7f511d001700 10 monclient(hunting): authenticate will time out at 5533.184649s 2023-09-30T00:51:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f511759e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f511813ac20 0x7f511814dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f5116d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51180a5fb0 0x7f511813e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f511659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51180b1460 0x7f511814d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f511759e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f511813ac20 0x7f511814dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60944/0 (socket says 172.21.15.40:60944) 2023-09-30T00:51:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f511759e700 1 -- 172.21.15.40:0/2798148578 learned_addr learned my addr 172.21.15.40:0/2798148578 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f511759e700 10 monclient(hunting): get_auth_request con 0x7f511813ac20 auth_method 0 2023-09-30T00:51:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f511759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f511759e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f511759e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f5116d9d700 10 monclient(hunting): get_auth_request con 0x7f51180a5fb0 auth_method 0 2023-09-30T00:51:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f5116d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f5116d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:11.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f5116d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f511659c700 10 monclient(hunting): get_auth_request con 0x7f51180b1460 auth_method 0 2023-09-30T00:51:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f511659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f511659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.884+0000 7f511659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.885+0000 7f511759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.885+0000 7f511759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.885+0000 7f511759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.885+0000 7f5116d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.885+0000 7f5116d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.885+0000 7f5116d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.885+0000 7f511659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.885+0000 7f511659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.885+0000 7f511659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.888+0000 7f511759e700 10 monclient(hunting): handle_auth_done global_id 4550 payload 824 2023-09-30T00:51:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.888+0000 7f511759e700 1 -- 172.21.15.40:0/2798148578 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51180a5fb0 msgr2=0x7f511813e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.888+0000 7f511759e700 1 --2- 172.21.15.40:0/2798148578 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51180a5fb0 0x7f511813e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.888+0000 7f511759e700 1 -- 172.21.15.40:0/2798148578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51180b1460 msgr2=0x7f511814d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.888+0000 7f511759e700 1 --2- 172.21.15.40:0/2798148578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51180b1460 0x7f511814d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.888+0000 7f511759e700 10 monclient: _finish_hunting 0 2023-09-30T00:51:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.888+0000 7f511759e700 1 monclient: found mon.noname-c 2023-09-30T00:51:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.888+0000 7f511759e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.888+0000 7f511759e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:51:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.888+0000 7f511759e700 1 -- 172.21.15.40:0/2798148578 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5110009a20 con 0x7f511813ac20 2023-09-30T00:51:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.891+0000 7f511759e700 1 --2- 172.21.15.40:0/2798148578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f511813ac20 0x7f511814dae0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f51180a7610 tx=0x7f5110019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.891+0000 7f50feffd700 10 client.?.objecter ms_handle_connect 0x7f511813ac20 2023-09-30T00:51:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.891+0000 7f50feffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:11.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.895+0000 7f50feffd700 1 -- 172.21.15.40:0/2798148578 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5110012ea0 con 0x7f511813ac20 2023-09-30T00:51:11.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.895+0000 7f50feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:11.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.895+0000 7f50feffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:51:11.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.895+0000 7f50feffd700 10 monclient: dump: 2023-09-30T00:51:11.897 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:11.897 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:11.897 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:11.897 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:11.897 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:11.897 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:11.897 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:11.898 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f50feffd700 10 monclient: _finish_auth 0 2023-09-30T00:51:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f50feffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f50feffd700 1 -- 172.21.15.40:0/2798148578 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5110027470 con 0x7f511813ac20 2023-09-30T00:51:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f511d001700 5 monclient: authenticate success, global_id 4550 2023-09-30T00:51:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f50feffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f50feffd700 1 -- 172.21.15.40:0/2798148578 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5110027850 con 0x7f511813ac20 2023-09-30T00:51:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f511d001700 10 monclient: _renew_subs 2023-09-30T00:51:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f511d001700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f511d001700 1 -- 172.21.15.40:0/2798148578 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51180a9940 con 0x7f511813ac20 2023-09-30T00:51:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f50feffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f50feffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:51:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f50feffd700 10 monclient: dump: 2023-09-30T00:51:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:11.900 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f511559a700 4 set_mon_vals no callback set 2023-09-30T00:51:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f511d001700 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f511d001700 10 monclient: _renew_subs 2023-09-30T00:51:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f511d001700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f511d001700 1 -- 172.21.15.40:0/2798148578 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51180aa4f0 con 0x7f511813ac20 2023-09-30T00:51:11.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.896+0000 7f511d001700 1 librados: init done 2023-09-30T00:51:11.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.897+0000 7f511d001700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:11.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.897+0000 7f511d001700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:11.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.897+0000 7f511d001700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:11.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:11.897+0000 7f511d001700 1 -- 172.21.15.40:0/2798148578 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51180a9f80 con 0x7f511813ac20 2023-09-30T00:51:12.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.085+0000 7f50feffd700 1 -- 172.21.15.40:0/2798148578 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5110024070 con 0x7f511813ac20 2023-09-30T00:51:12.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.085+0000 7f50feffd700 1 --2- 172.21.15.40:0/2798148578 >> v2:172.21.15.135:6800/116426 conn(0x7f5104068890 0x7f510406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:12.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.085+0000 7f50feffd700 1 -- 172.21.15.40:0/2798148578 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5110086ac0 con 0x7f511813ac20 2023-09-30T00:51:12.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.085+0000 7f50feffd700 10 client.4550.objecter ms_dispatch 0x7f5118001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:12.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.085+0000 7f50feffd700 3 client.4550.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:12.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.085+0000 7f50feffd700 3 client.4550.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.085+0000 7f50feffd700 20 client.4550.objecter dump_active .. 0 homeless 2023-09-30T00:51:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.085+0000 7f5116d9d700 1 --2- 172.21.15.40:0/2798148578 >> v2:172.21.15.135:6800/116426 conn(0x7f5104068890 0x7f510406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.085+0000 7f5116d9d700 10 monclient: get_auth_request con 0x7f5104068890 auth_method 0 2023-09-30T00:51:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.086+0000 7f50feffd700 10 client.4550.objecter ms_handle_connect 0x7f5104068890 2023-09-30T00:51:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.086+0000 7f5116d9d700 1 --2- 172.21.15.40:0/2798148578 >> v2:172.21.15.135:6800/116426 conn(0x7f5104068890 0x7f510406ad40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5108009db0 tx=0x7f51080091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.193+0000 7f50feffd700 1 -- 172.21.15.40:0/2798148578 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f51100543e0 con 0x7f511813ac20 2023-09-30T00:51:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.193+0000 7f50feffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.193+0000 7f50feffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:12.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.509+0000 7f511d001700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:12.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.509+0000 7f511d001700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:12.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.509+0000 7f511d001700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:12.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.510+0000 7f511d001700 1 -- 172.21.15.40:0/2798148578 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f51180a9f80 con 0x7f511813ac20 2023-09-30T00:51:12.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.521+0000 7f50feffd700 1 -- 172.21.15.40:0/2798148578 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f51180a9f80 con 0x7f511813ac20 2023-09-30T00:51:12.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.521+0000 7f50feffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:12.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.521+0000 7f50feffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:12.522 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:12.522 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:12.524 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.523+0000 7f511d001700 10 librados: watch_flush enter 2023-09-30T00:51:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.523+0000 7f511d001700 10 librados: watch_flush exit 2023-09-30T00:51:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.523+0000 7f511d001700 20 client.4550.objecter shutdown clearing up homeless session... 2023-09-30T00:51:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.523+0000 7f511d001700 10 client.4550.objecter successfully canceled tick 2023-09-30T00:51:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 1 -- 172.21.15.40:0/2798148578 >> v2:172.21.15.135:6800/116426 conn(0x7f5104068890 msgr2=0x7f510406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 1 --2- 172.21.15.40:0/2798148578 >> v2:172.21.15.135:6800/116426 conn(0x7f5104068890 0x7f510406ad40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5108009db0 tx=0x7f51080091a0 comp rx=0 tx=0).stop 2023-09-30T00:51:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 10 monclient: shutdown 2023-09-30T00:51:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 1 -- 172.21.15.40:0/2798148578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f511813ac20 msgr2=0x7f511814dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 1 --2- 172.21.15.40:0/2798148578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f511813ac20 0x7f511814dae0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f51180a7610 tx=0x7f5110019000 comp rx=0 tx=0).stop 2023-09-30T00:51:12.525 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 1 -- 172.21.15.40:0/2798148578 shutdown_connections 2023-09-30T00:51:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 1 --2- 172.21.15.40:0/2798148578 >> v2:172.21.15.135:6800/116426 conn(0x7f5104068890 0x7f510406ad40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 1 --2- 172.21.15.40:0/2798148578 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f51180a5fb0 0x7f511813e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 1 --2- 172.21.15.40:0/2798148578 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f511813ac20 0x7f511814dae0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 1 --2- 172.21.15.40:0/2798148578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51180b1460 0x7f511814d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 1 -- 172.21.15.40:0/2798148578 >> 172.21.15.40:0/2798148578 conn(0x7f51180631f0 msgr2=0x7f51180afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 1 -- 172.21.15.40:0/2798148578 shutdown_connections 2023-09-30T00:51:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 1 -- 172.21.15.40:0/2798148578 wait complete. 2023-09-30T00:51:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:12.524+0000 7f511d001700 1 librados: shutdown 2023-09-30T00:51:13.654 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:04:49.365 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:13.658 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:04:49.370 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:13.693 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:04:49.404 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:13.711 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:04:49.397 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:13.713 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:04:49.400 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:13.725 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:04:49.436 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:13.728 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:04:49.440 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:13.742 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:04:49.429 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:13.761 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:04:49.473 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:15.540 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:15.680 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.680+0000 7f0c74cd8700 10 monclient: get_monmap_and_config 2023-09-30T00:51:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.680+0000 7f0c74cd8700 10 monclient: build_initial_monmap 2023-09-30T00:51:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.680+0000 7f0c74cd8700 10 monclient: monmap: 2023-09-30T00:51:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:15.681150+0000 2023-09-30T00:51:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:15.681150+0000 2023-09-30T00:51:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:15.681 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:15.682 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.680+0000 7f0c74cd8700 1 Processor -- start 2023-09-30T00:51:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.680+0000 7f0c74cd8700 1 -- start start 2023-09-30T00:51:15.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.680+0000 7f0c74cd8700 10 monclient: init 2023-09-30T00:51:15.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:15.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-09-30T00:51:15.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c700b1460 0x7f0c700b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 10 monclient(hunting): picked mon.noname-b con 0x7f0c700b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0c700a5fb0 0x7f0c70141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 10 monclient(hunting): picked mon.noname-a con 0x7f0c700a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:51:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0c700af410 0x7f0c700af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 10 monclient(hunting): picked mon.noname-c con 0x7f0c700af410 addr v2:172.21.15.135:3300/0 2023-09-30T00:51:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c700a71f0 con 0x7f0c700a5fb0 2023-09-30T00:51:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c700a74d0 con 0x7f0c700af410 2023-09-30T00:51:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700a63c0 con 0x7f0c700b1460 2023-09-30T00:51:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:15.685 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.681+0000 7f0c74cd8700 10 monclient(hunting): authenticate will time out at 5536.982695s 2023-09-30T00:51:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6dd9b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0c700a5fb0 0x7f0c70141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6e59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c700b1460 0x7f0c700b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6ed9d700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f0c700af410 0x7f0c700af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6e59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c700b1460 0x7f0c700b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60962/0 (socket says 172.21.15.40:60962) 2023-09-30T00:51:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6e59c700 1 -- 172.21.15.40:0/790401882 learned_addr learned my addr 172.21.15.40:0/790401882 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6e59c700 10 monclient(hunting): get_auth_request con 0x7f0c700b1460 auth_method 0 2023-09-30T00:51:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6dd9b700 10 monclient(hunting): get_auth_request con 0x7f0c700a5fb0 auth_method 0 2023-09-30T00:51:15.686 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:15.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:15.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:15.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6ed9d700 10 monclient(hunting): get_auth_request con 0x7f0c700af410 auth_method 0 2023-09-30T00:51:15.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:15.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:15.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.682+0000 7f0c6ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:15.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.683+0000 7f0c6ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:15.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.683+0000 7f0c6ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:15.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.683+0000 7f0c6ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:15.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.683+0000 7f0c6e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:15.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.683+0000 7f0c6e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:15.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.683+0000 7f0c6e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:15.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.683+0000 7f0c6dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:15.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.683+0000 7f0c6dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:15.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.683+0000 7f0c6dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:15.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.686+0000 7f0c6ed9d700 10 monclient(hunting): handle_auth_done global_id 4684 payload 274 2023-09-30T00:51:15.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.686+0000 7f0c6ed9d700 1 -- 172.21.15.40:0/790401882 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c700b1460 msgr2=0x7f0c700b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:15.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.686+0000 7f0c6ed9d700 1 --2- 172.21.15.40:0/790401882 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c700b1460 0x7f0c700b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:15.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.686+0000 7f0c6ed9d700 1 -- 172.21.15.40:0/790401882 >> v2:172.21.15.40:3300/0 conn(0x7f0c700a5fb0 msgr2=0x7f0c70141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:15.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.686+0000 7f0c6ed9d700 1 --2- 172.21.15.40:0/790401882 >> v2:172.21.15.40:3300/0 conn(0x7f0c700a5fb0 0x7f0c70141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:15.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.686+0000 7f0c6ed9d700 10 monclient: _finish_hunting 0 2023-09-30T00:51:15.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.686+0000 7f0c6ed9d700 1 monclient: found mon.noname-c 2023-09-30T00:51:15.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.686+0000 7f0c6ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:15.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.686+0000 7f0c6ed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:51:15.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.686+0000 7f0c6ed9d700 1 -- 172.21.15.40:0/790401882 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c700a56d0 con 0x7f0c700af410 2023-09-30T00:51:15.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.688+0000 7f0c6ed9d700 1 --2- 172.21.15.40:0/790401882 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c700af410 0x7f0c700af890 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0c700b05f0 tx=0x7f0c68009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5a749171cc5b3c3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c6d59a700 1 -- 172.21.15.40:0/790401882 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6800e070 con 0x7f0c700af410 2023-09-30T00:51:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c6d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c6d59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:51:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c6d59a700 10 monclient: dump: 2023-09-30T00:51:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:15.694 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:15.695 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c6d59a700 10 monclient: _finish_auth 0 2023-09-30T00:51:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c6d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c6d59a700 1 -- 172.21.15.40:0/790401882 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6800bab0 con 0x7f0c700af410 2023-09-30T00:51:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c74cd8700 5 monclient: authenticate success, global_id 4684 2023-09-30T00:51:15.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c74cd8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:51:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c6d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c74cd8700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c6d59a700 1 -- 172.21.15.40:0/790401882 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6800be70 con 0x7f0c700af410 2023-09-30T00:51:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c74cd8700 4 set_mon_vals no callback set 2023-09-30T00:51:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c6d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c6d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.692+0000 7f0c6d59a700 10 monclient: dump: 2023-09-30T00:51:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:15.696 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:15.697 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.693+0000 7f0c74cd8700 10 monclient: shutdown 2023-09-30T00:51:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.693+0000 7f0c74cd8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.693+0000 7f0c74cd8700 1 -- 172.21.15.40:0/790401882 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c700af410 msgr2=0x7f0c700af890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.693+0000 7f0c74cd8700 1 --2- 172.21.15.40:0/790401882 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c700af410 0x7f0c700af890 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0c700b05f0 tx=0x7f0c68009780 comp rx=0 tx=0).stop 2023-09-30T00:51:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.693+0000 7f0c74cd8700 1 -- 172.21.15.40:0/790401882 shutdown_connections 2023-09-30T00:51:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.693+0000 7f0c74cd8700 1 --2- 172.21.15.40:0/790401882 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c700af410 0x7f0c700af890 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:15.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.693+0000 7f0c74cd8700 1 --2- 172.21.15.40:0/790401882 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c700b1460 0x7f0c700b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.693+0000 7f0c74cd8700 1 --2- 172.21.15.40:0/790401882 >> v2:172.21.15.40:3300/0 conn(0x7f0c700a5fb0 0x7f0c70141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.693+0000 7f0c74cd8700 1 -- 172.21.15.40:0/790401882 >> 172.21.15.40:0/790401882 conn(0x7f0c700631f0 msgr2=0x7f0c700635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.693+0000 7f0c74cd8700 1 -- 172.21.15.40:0/790401882 shutdown_connections 2023-09-30T00:51:15.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.693+0000 7f0c74cd8700 1 -- 172.21.15.40:0/790401882 wait complete. 2023-09-30T00:51:15.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.694+0000 7f0c74cd8700 10 monclient: build_initial_monmap 2023-09-30T00:51:15.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.694+0000 7f0c74cd8700 10 monclient: monmap: 2023-09-30T00:51:15.700 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:15.701 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:15.701 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:15.701 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:15.701 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:15.701 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:15.701 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:15.701 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:15.701 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:15.701 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:15.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.694+0000 7f0c74cd8700 1 librados: starting msgr at 2023-09-30T00:51:15.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.694+0000 7f0c74cd8700 1 librados: starting objecter 2023-09-30T00:51:15.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 1 Processor -- start 2023-09-30T00:51:15.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 1 -- start start 2023-09-30T00:51:15.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 1 librados: setting wanted keys 2023-09-30T00:51:15.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 1 librados: calling monclient init 2023-09-30T00:51:15.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 10 monclient: init 2023-09-30T00:51:15.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:15.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:51:15.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c700a5fb0 0x7f0c70144540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:15.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 10 monclient(hunting): picked mon.noname-c con 0x7f0c700a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:15.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c700af410 0x7f0c7014e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:15.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 10 monclient(hunting): picked mon.noname-b con 0x7f0c700af410 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:15.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c700b1460 0x7f0c7013c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 10 monclient(hunting): picked mon.noname-a con 0x7f0c700b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c700a81a0 con 0x7f0c700b1460 2023-09-30T00:51:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7014e930 con 0x7f0c700af410 2023-09-30T00:51:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7014eaf0 con 0x7f0c700a5fb0 2023-09-30T00:51:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.695+0000 7f0c74cd8700 10 monclient(hunting): authenticate will time out at 5536.996777s 2023-09-30T00:51:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.696+0000 7f0c6e59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c700a5fb0 0x7f0c70144540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.696+0000 7f0c6ed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c700b1460 0x7f0c7013c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.696+0000 7f0c6dd9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c700af410 0x7f0c7014e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.696+0000 7f0c6ed9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c700b1460 0x7f0c7013c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42284/0 (socket says 172.21.15.40:42284) 2023-09-30T00:51:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.696+0000 7f0c6e59c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c700a5fb0 0x7f0c70144540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:60964/0 (socket says 172.21.15.40:60964) 2023-09-30T00:51:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.696+0000 7f0c6ed9d700 1 -- 172.21.15.40:0/2959427474 learned_addr learned my addr 172.21.15.40:0/2959427474 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.696+0000 7f0c6ed9d700 10 monclient(hunting): get_auth_request con 0x7f0c700b1460 auth_method 0 2023-09-30T00:51:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.696+0000 7f0c6ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.696+0000 7f0c6ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.696+0000 7f0c6ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.696+0000 7f0c6e59c700 10 monclient(hunting): get_auth_request con 0x7f0c700a5fb0 auth_method 0 2023-09-30T00:51:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.696+0000 7f0c6e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.696+0000 7f0c6e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.696+0000 7f0c6e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.697+0000 7f0c6dd9b700 10 monclient(hunting): get_auth_request con 0x7f0c700af410 auth_method 0 2023-09-30T00:51:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.697+0000 7f0c6dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.697+0000 7f0c6dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.697+0000 7f0c6dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.697+0000 7f0c6ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.697+0000 7f0c6ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.697+0000 7f0c6ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.697+0000 7f0c6e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.697+0000 7f0c6e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.697+0000 7f0c6e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.697+0000 7f0c6dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.697+0000 7f0c6dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.697+0000 7f0c6dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.701+0000 7f0c6dd9b700 10 monclient(hunting): handle_auth_done global_id 4687 payload 824 2023-09-30T00:51:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.701+0000 7f0c6dd9b700 1 -- 172.21.15.40:0/2959427474 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c700a5fb0 msgr2=0x7f0c70144540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.701+0000 7f0c6dd9b700 1 --2- 172.21.15.40:0/2959427474 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c700a5fb0 0x7f0c70144540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.701+0000 7f0c6dd9b700 1 -- 172.21.15.40:0/2959427474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c700b1460 msgr2=0x7f0c7013c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.701+0000 7f0c6dd9b700 1 --2- 172.21.15.40:0/2959427474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c700b1460 0x7f0c7013c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.701+0000 7f0c6dd9b700 10 monclient: _finish_hunting 0 2023-09-30T00:51:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.701+0000 7f0c6dd9b700 1 monclient: found mon.noname-b 2023-09-30T00:51:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.701+0000 7f0c6dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.701+0000 7f0c6dd9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:51:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.701+0000 7f0c6dd9b700 1 -- 172.21.15.40:0/2959427474 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c68009a20 con 0x7f0c700af410 2023-09-30T00:51:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.703+0000 7f0c6dd9b700 1 --2- 172.21.15.40:0/2959427474 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c700af410 0x7f0c7014e3f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800ea50 tx=0x7f0c580091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.703+0000 7f0c667fc700 10 client.?.objecter ms_handle_connect 0x7f0c700af410 2023-09-30T00:51:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.703+0000 7f0c667fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:15.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c667fc700 1 -- 172.21.15.40:0/2959427474 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5800ee90 con 0x7f0c700af410 2023-09-30T00:51:15.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:15.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c667fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:15.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c667fc700 10 monclient: dump: 2023-09-30T00:51:15.709 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:15.710 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c667fc700 10 monclient: _finish_auth 0 2023-09-30T00:51:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c667fc700 1 -- 172.21.15.40:0/2959427474 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5800c7f0 con 0x7f0c700af410 2023-09-30T00:51:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c667fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c667fc700 1 -- 172.21.15.40:0/2959427474 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5800cb90 con 0x7f0c700af410 2023-09-30T00:51:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c74cd8700 5 monclient: authenticate success, global_id 4687 2023-09-30T00:51:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c74cd8700 10 monclient: _renew_subs 2023-09-30T00:51:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c74cd8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c74cd8700 1 -- 172.21.15.40:0/2959427474 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7013c8c0 con 0x7f0c700af410 2023-09-30T00:51:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c6cd99700 4 set_mon_vals no callback set 2023-09-30T00:51:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c667fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c667fc700 10 monclient: dump: 2023-09-30T00:51:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:15.712 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c74cd8700 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c74cd8700 10 monclient: _renew_subs 2023-09-30T00:51:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c74cd8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c74cd8700 1 -- 172.21.15.40:0/2959427474 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c7013ce30 con 0x7f0c700af410 2023-09-30T00:51:15.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.707+0000 7f0c74cd8700 1 librados: init done 2023-09-30T00:51:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.708+0000 7f0c74cd8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.708+0000 7f0c74cd8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.708+0000 7f0c74cd8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:15.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.708+0000 7f0c74cd8700 1 -- 172.21.15.40:0/2959427474 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7014ecb0 con 0x7f0c700af410 2023-09-30T00:51:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.727+0000 7f0c667fc700 1 -- 172.21.15.40:0/2959427474 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0c58013470 con 0x7f0c700af410 2023-09-30T00:51:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.728+0000 7f0c667fc700 1 --2- 172.21.15.40:0/2959427474 >> v2:172.21.15.135:6800/116426 conn(0x7f0c50068890 0x7f0c5006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.728+0000 7f0c6e59c700 1 --2- 172.21.15.40:0/2959427474 >> v2:172.21.15.135:6800/116426 conn(0x7f0c50068890 0x7f0c5006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.728+0000 7f0c6e59c700 10 monclient: get_auth_request con 0x7f0c50068890 auth_method 0 2023-09-30T00:51:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.729+0000 7f0c6e59c700 1 --2- 172.21.15.40:0/2959427474 >> v2:172.21.15.135:6800/116426 conn(0x7f0c50068890 0x7f0c5006ad40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0c60006fd0 tx=0x7f0c60009250 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.729+0000 7f0c667fc700 10 client.4687.objecter ms_handle_connect 0x7f0c50068890 2023-09-30T00:51:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.964+0000 7f0c667fc700 1 -- 172.21.15.40:0/2959427474 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0c580564c0 con 0x7f0c700af410 2023-09-30T00:51:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.964+0000 7f0c667fc700 10 client.4687.objecter ms_dispatch 0x7f0c70001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.964+0000 7f0c667fc700 3 client.4687.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.964+0000 7f0c667fc700 3 client.4687.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.964+0000 7f0c667fc700 20 client.4687.objecter dump_active .. 0 homeless 2023-09-30T00:51:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.976+0000 7f0c667fc700 1 -- 172.21.15.40:0/2959427474 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c58052100 con 0x7f0c700af410 2023-09-30T00:51:15.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.976+0000 7f0c667fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:15.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:15.976+0000 7f0c667fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:16.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.255+0000 7f0c74cd8700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:16.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.255+0000 7f0c74cd8700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:16.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.255+0000 7f0c74cd8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:16.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.255+0000 7f0c74cd8700 1 -- 172.21.15.40:0/2959427474 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0c70064a00 con 0x7f0c700af410 2023-09-30T00:51:16.261 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.261+0000 7f0c667fc700 1 -- 172.21.15.40:0/2959427474 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0c58017100 con 0x7f0c700af410 2023-09-30T00:51:16.261 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.261+0000 7f0c667fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:16.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.261+0000 7f0c667fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:16.262 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:16.262 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:16.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 10 librados: watch_flush enter 2023-09-30T00:51:16.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 10 librados: watch_flush exit 2023-09-30T00:51:16.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 20 client.4687.objecter shutdown clearing up homeless session... 2023-09-30T00:51:16.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 10 client.4687.objecter successfully canceled tick 2023-09-30T00:51:16.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 1 -- 172.21.15.40:0/2959427474 >> v2:172.21.15.135:6800/116426 conn(0x7f0c50068890 msgr2=0x7f0c5006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:16.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 1 --2- 172.21.15.40:0/2959427474 >> v2:172.21.15.135:6800/116426 conn(0x7f0c50068890 0x7f0c5006ad40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0c60006fd0 tx=0x7f0c60009250 comp rx=0 tx=0).stop 2023-09-30T00:51:16.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 10 monclient: shutdown 2023-09-30T00:51:16.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:16.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 1 -- 172.21.15.40:0/2959427474 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c700af410 msgr2=0x7f0c7014e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:16.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 1 --2- 172.21.15.40:0/2959427474 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c700af410 0x7f0c7014e3f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800ea50 tx=0x7f0c580091a0 comp rx=0 tx=0).stop 2023-09-30T00:51:16.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 1 -- 172.21.15.40:0/2959427474 shutdown_connections 2023-09-30T00:51:16.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 1 --2- 172.21.15.40:0/2959427474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c700b1460 0x7f0c7013c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:16.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 1 --2- 172.21.15.40:0/2959427474 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f0c700a5fb0 0x7f0c70144540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:16.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 1 --2- 172.21.15.40:0/2959427474 >> v2:172.21.15.135:6800/116426 conn(0x7f0c50068890 0x7f0c5006ad40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:16.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 1 --2- 172.21.15.40:0/2959427474 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f0c700af410 0x7f0c7014e3f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:16.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 1 -- 172.21.15.40:0/2959427474 >> 172.21.15.40:0/2959427474 conn(0x7f0c700631f0 msgr2=0x7f0c700b28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:16.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 1 -- 172.21.15.40:0/2959427474 shutdown_connections 2023-09-30T00:51:16.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 1 -- 172.21.15.40:0/2959427474 wait complete. 2023-09-30T00:51:16.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:16.263+0000 7f0c74cd8700 1 librados: shutdown 2023-09-30T00:51:19.278 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:19.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.413+0000 7f7b0fe9a700 10 monclient: get_monmap_and_config 2023-09-30T00:51:19.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.413+0000 7f7b0fe9a700 10 monclient: build_initial_monmap 2023-09-30T00:51:19.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.413+0000 7f7b0fe9a700 10 monclient: monmap: 2023-09-30T00:51:19.414 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:19.414 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:19.415 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:19.414718+0000 2023-09-30T00:51:19.415 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:19.414718+0000 2023-09-30T00:51:19.415 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:19.415 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:19.415 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:19.415 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:19.415 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:19.415 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:19.415 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:19.415 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.414+0000 7f7b0fe9a700 1 Processor -- start 2023-09-30T00:51:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.414+0000 7f7b0fe9a700 1 -- start start 2023-09-30T00:51:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.414+0000 7f7b0fe9a700 10 monclient: init 2023-09-30T00:51:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.415+0000 7f7b0fe9a700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.415+0000 7f7b0fe9a700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-09-30T00:51:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.415+0000 7f7b0fe9a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b080b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:51:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.415+0000 7f7b0fe9a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b080b1830 addr v1:172.21.15.135:6789/0 2023-09-30T00:51:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.415+0000 7f7b0fe9a700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7b080a5fb0 0x7f7b0813b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:19.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.415+0000 7f7b0fe9a700 10 monclient(hunting): picked mon.noname-c con 0x7f7b080a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:51:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.415+0000 7f7b0fe9a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b080a5b20 con 0x7f7b080b1460 2023-09-30T00:51:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.415+0000 7f7b0fe9a700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b080a71f0 con 0x7f7b080b1830 2023-09-30T00:51:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.415+0000 7f7b0fe9a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.415+0000 7f7b0fe9a700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b08141e40 con 0x7f7b080a5fb0 2023-09-30T00:51:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.415+0000 7f7b0fe9a700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.415+0000 7f7b0fe9a700 10 monclient(hunting): authenticate will time out at 5540.716200s 2023-09-30T00:51:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.415+0000 7f7b0dc36700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7b080b1460 0x7f7b08141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37926/0 (socket says 172.21.15.40:37926) 2023-09-30T00:51:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.415+0000 7f7b0dc36700 1 -- 172.21.15.40:0/3564893910 learned_addr learned my addr 172.21.15.40:0/3564893910 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:51:19.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.416+0000 7f7b0e437700 1 --2- 172.21.15.40:0/3564893910 >> v2:172.21.15.135:3300/0 conn(0x7f7b080a5fb0 0x7f7b0813b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.417+0000 7f7b0e437700 10 monclient(hunting): get_auth_request con 0x7f7b080a5fb0 auth_method 0 2023-09-30T00:51:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.417+0000 7f7b0e437700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.417+0000 7f7b0e437700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.417+0000 7f7b0e437700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.418+0000 7f7b0e437700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.418+0000 7f7b0e437700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:19.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.419+0000 7f7b0e437700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.419+0000 7f7b0cc34700 1 -- 172.21.15.40:0/3564893910 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375425338 0 0) 0x7f7b080a5b20 con 0x7f7b080b1460 2023-09-30T00:51:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.419+0000 7f7b0cc34700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.419+0000 7f7b0cc34700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.419+0000 7f7b0cc34700 10 monclient(hunting): my global_id is 4626 2023-09-30T00:51:19.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.419+0000 7f7b0cc34700 1 -- 172.21.15.40:0/3564893910 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7af4003570 con 0x7f7b080b1460 2023-09-30T00:51:19.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.423+0000 7f7b0e437700 10 monclient(hunting): handle_auth_done global_id 4690 payload 274 2023-09-30T00:51:19.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.423+0000 7f7b0e437700 1 -- 172.21.15.40:0/3564893910 >> v1:172.21.15.135:6789/0 conn(0x7f7b080b1830 legacy=0x7f7b0813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:19.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.423+0000 7f7b0e437700 1 -- 172.21.15.40:0/3564893910 >> v1:172.21.15.40:6789/0 conn(0x7f7b080b1460 legacy=0x7f7b08141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:19.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.423+0000 7f7b0e437700 10 monclient: _finish_hunting 0 2023-09-30T00:51:19.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.423+0000 7f7b0e437700 1 monclient: found mon.noname-c 2023-09-30T00:51:19.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.423+0000 7f7b0e437700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:19.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.423+0000 7f7b0e437700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:51:19.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.423+0000 7f7b0e437700 1 -- 172.21.15.40:0/3564893910 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b080a74d0 con 0x7f7b080a5fb0 2023-09-30T00:51:19.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.428+0000 7f7b0e437700 1 --2- 172.21.15.40:0/3564893910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b080a5fb0 0x7f7b0813b330 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7b00009e90 tx=0x7f7b00009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da89f154c261b23a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0cc34700 1 -- 172.21.15.40:0/3564893910 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0000e070 con 0x7f7b080a5fb0 2023-09-30T00:51:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0cc34700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0cc34700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:51:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0cc34700 10 monclient: dump: 2023-09-30T00:51:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:19.434 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:19.435 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:19.435 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:19.435 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:19.435 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:19.435 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:19.435 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:19.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0cc34700 10 monclient: _finish_auth 0 2023-09-30T00:51:19.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0cc34700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:19.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0cc34700 1 -- 172.21.15.40:0/3564893910 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0000bc40 con 0x7f7b080a5fb0 2023-09-30T00:51:19.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0cc34700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:19.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0cc34700 1 -- 172.21.15.40:0/3564893910 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b00012430 con 0x7f7b080a5fb0 2023-09-30T00:51:19.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0fe9a700 5 monclient: authenticate success, global_id 4690 2023-09-30T00:51:19.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0fe9a700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:19.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0cc34700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:19.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0fe9a700 4 set_mon_vals no callback set 2023-09-30T00:51:19.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0cc34700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:19.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.433+0000 7f7b0cc34700 10 monclient: dump: 2023-09-30T00:51:19.436 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:19.436 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:19.436 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:19.437 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:19.437 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:19.437 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:19.437 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:19.437 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:19.437 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:19.437 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:19.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.434+0000 7f7b0fe9a700 10 monclient: shutdown 2023-09-30T00:51:19.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.434+0000 7f7b0fe9a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:19.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.434+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/3564893910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b080a5fb0 msgr2=0x7f7b0813b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:19.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.434+0000 7f7b0fe9a700 1 --2- 172.21.15.40:0/3564893910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b080a5fb0 0x7f7b0813b330 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7b00009e90 tx=0x7f7b00009780 comp rx=0 tx=0).stop 2023-09-30T00:51:19.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.434+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/3564893910 shutdown_connections 2023-09-30T00:51:19.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.434+0000 7f7b0fe9a700 1 --2- 172.21.15.40:0/3564893910 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b080a5fb0 0x7f7b0813b330 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:19.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.434+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/3564893910 >> 172.21.15.40:0/3564893910 conn(0x7f7b080631f0 msgr2=0x7f7b080635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:19.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.434+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/3564893910 shutdown_connections 2023-09-30T00:51:19.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.434+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/3564893910 wait complete. 2023-09-30T00:51:19.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.435+0000 7f7b0fe9a700 10 monclient: build_initial_monmap 2023-09-30T00:51:19.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.435+0000 7f7b0fe9a700 10 monclient: monmap: 2023-09-30T00:51:19.441 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:19.441 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:19.441 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:19.441 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:19.442 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.435+0000 7f7b0fe9a700 1 librados: starting msgr at 2023-09-30T00:51:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.435+0000 7f7b0fe9a700 1 librados: starting objecter 2023-09-30T00:51:19.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.435+0000 7f7b0fe9a700 1 Processor -- start 2023-09-30T00:51:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.435+0000 7f7b0fe9a700 1 -- start start 2023-09-30T00:51:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.435+0000 7f7b0fe9a700 1 librados: setting wanted keys 2023-09-30T00:51:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.435+0000 7f7b0fe9a700 1 librados: calling monclient init 2023-09-30T00:51:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.435+0000 7f7b0fe9a700 10 monclient: init 2023-09-30T00:51:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:51:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b080a5fb0 0x7f7b080a8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 10 monclient(hunting): picked mon.noname-c con 0x7f7b080a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:19.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b080b1460 0x7f7b0813e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 10 monclient(hunting): picked mon.noname-b con 0x7f7b080b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b080b1830 0x7f7b080a9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 10 monclient(hunting): picked mon.noname-a con 0x7f7b080b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b080a7f10 con 0x7f7b080b1830 2023-09-30T00:51:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0813efd0 con 0x7f7b080b1460 2023-09-30T00:51:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0813e730 con 0x7f7b080a5fb0 2023-09-30T00:51:19.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:19.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0fe9a700 10 monclient(hunting): authenticate will time out at 5540.737303s 2023-09-30T00:51:19.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0e437700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b080b1830 0x7f7b080a9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:19.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0dc36700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b080a5fb0 0x7f7b080a8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:19.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.436+0000 7f7b0d435700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b080b1460 0x7f7b0813e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:19.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0e437700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b080b1830 0x7f7b080a9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42288/0 (socket says 172.21.15.40:42288) 2023-09-30T00:51:19.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0e437700 1 -- 172.21.15.40:0/407690790 learned_addr learned my addr 172.21.15.40:0/407690790 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:19.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0e437700 10 monclient(hunting): get_auth_request con 0x7f7b080b1830 auth_method 0 2023-09-30T00:51:19.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0e437700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:19.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0e437700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0e437700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0d435700 10 monclient(hunting): get_auth_request con 0x7f7b080b1460 auth_method 0 2023-09-30T00:51:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0d435700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0d435700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0d435700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0dc36700 10 monclient(hunting): get_auth_request con 0x7f7b080a5fb0 auth_method 0 2023-09-30T00:51:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0dc36700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0dc36700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0dc36700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:19.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0d435700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:19.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0d435700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:19.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.437+0000 7f7b0d435700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:19.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.438+0000 7f7b0e437700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:19.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.438+0000 7f7b0e437700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:19.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.438+0000 7f7b0e437700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:19.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.438+0000 7f7b0dc36700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:19.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.438+0000 7f7b0dc36700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:19.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.438+0000 7f7b0dc36700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.441+0000 7f7b0d435700 10 monclient(hunting): handle_auth_done global_id 4696 payload 824 2023-09-30T00:51:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.441+0000 7f7b0d435700 1 -- 172.21.15.40:0/407690790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b080a5fb0 msgr2=0x7f7b080a8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.441+0000 7f7b0d435700 1 --2- 172.21.15.40:0/407690790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b080a5fb0 0x7f7b080a8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:19.449 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.441+0000 7f7b0d435700 1 -- 172.21.15.40:0/407690790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b080b1830 msgr2=0x7f7b080a9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.442+0000 7f7b0d435700 1 --2- 172.21.15.40:0/407690790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b080b1830 0x7f7b080a9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.442+0000 7f7b0d435700 10 monclient: _finish_hunting 0 2023-09-30T00:51:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.442+0000 7f7b0d435700 1 monclient: found mon.noname-b 2023-09-30T00:51:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.442+0000 7f7b0d435700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.442+0000 7f7b0d435700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:51:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.442+0000 7f7b0d435700 1 -- 172.21.15.40:0/407690790 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b00009a20 con 0x7f7b080b1460 2023-09-30T00:51:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.444+0000 7f7b0d435700 1 --2- 172.21.15.40:0/407690790 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b080b1460 0x7f7b0813e1f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400c490 tx=0x7f7b04007ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.444+0000 7f7afdffb700 10 client.?.objecter ms_handle_connect 0x7f7b080b1460 2023-09-30T00:51:19.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.444+0000 7f7afdffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7afdffb700 1 -- 172.21.15.40:0/407690790 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0400ce40 con 0x7f7b080b1460 2023-09-30T00:51:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7afdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7afdffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7afdffb700 10 monclient: dump: 2023-09-30T00:51:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:19.451 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:19.452 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7afdffb700 10 monclient: _finish_auth 0 2023-09-30T00:51:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7afdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7afdffb700 1 -- 172.21.15.40:0/407690790 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b04011590 con 0x7f7b080b1460 2023-09-30T00:51:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7b0fe9a700 5 monclient: authenticate success, global_id 4696 2023-09-30T00:51:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7b0fe9a700 10 monclient: _renew_subs 2023-09-30T00:51:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7b0fe9a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:19.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/407690790 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b080ae5a0 con 0x7f7b080b1460 2023-09-30T00:51:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7afdffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7afdffb700 1 -- 172.21.15.40:0/407690790 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b04011950 con 0x7f7b080b1460 2023-09-30T00:51:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7afdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7afdffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7afdffb700 10 monclient: dump: 2023-09-30T00:51:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:19.453 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:19.454 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7afffff700 4 set_mon_vals no callback set 2023-09-30T00:51:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7b0fe9a700 10 client.4696.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7b0fe9a700 10 monclient: _renew_subs 2023-09-30T00:51:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7b0fe9a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:19.454 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/407690790 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b080a79f0 con 0x7f7b080b1460 2023-09-30T00:51:19.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.448+0000 7f7b0fe9a700 1 librados: init done 2023-09-30T00:51:19.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.449+0000 7f7b0fe9a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:19.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.449+0000 7f7b0fe9a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:19.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.449+0000 7f7b0fe9a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:19.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.449+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/407690790 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b08064a00 con 0x7f7b080b1460 2023-09-30T00:51:19.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.469+0000 7f7afdffb700 1 -- 172.21.15.40:0/407690790 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7b04011cc0 con 0x7f7b080b1460 2023-09-30T00:51:19.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.470+0000 7f7afdffb700 1 --2- 172.21.15.40:0/407690790 >> v2:172.21.15.135:6800/116426 conn(0x7f7aec068890 0x7f7aec06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:19.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.470+0000 7f7b0dc36700 1 --2- 172.21.15.40:0/407690790 >> v2:172.21.15.135:6800/116426 conn(0x7f7aec068890 0x7f7aec06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:19.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.470+0000 7f7b0dc36700 10 monclient: get_auth_request con 0x7f7aec068890 auth_method 0 2023-09-30T00:51:19.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.470+0000 7f7afdffb700 10 client.4696.objecter ms_handle_connect 0x7f7aec068890 2023-09-30T00:51:19.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.470+0000 7f7b0dc36700 1 --2- 172.21.15.40:0/407690790 >> v2:172.21.15.135:6800/116426 conn(0x7f7aec068890 0x7f7aec06ad40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7af8007fd0 tx=0x7f7af8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.476+0000 7f7afdffb700 1 -- 172.21.15.40:0/407690790 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7b040551c0 con 0x7f7b080b1460 2023-09-30T00:51:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.476+0000 7f7afdffb700 10 client.4696.objecter ms_dispatch 0x7f7b08001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.476+0000 7f7afdffb700 3 client.4696.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:19.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.476+0000 7f7afdffb700 3 client.4696.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:19.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.476+0000 7f7afdffb700 20 client.4696.objecter dump_active .. 0 homeless 2023-09-30T00:51:19.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.717+0000 7f7afdffb700 1 -- 172.21.15.40:0/407690790 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b04050e00 con 0x7f7b080b1460 2023-09-30T00:51:19.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.717+0000 7f7afdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:19.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.717+0000 7f7afdffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:19.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.973+0000 7f7b0fe9a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:19.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.973+0000 7f7b0fe9a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:19.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.973+0000 7f7b0fe9a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:19.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.973+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/407690790 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7b080a6410 con 0x7f7b080b1460 2023-09-30T00:51:19.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.978+0000 7f7afdffb700 1 -- 172.21.15.40:0/407690790 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7b04017070 con 0x7f7b080b1460 2023-09-30T00:51:19.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.978+0000 7f7afdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:19.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.978+0000 7f7afdffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:19.978 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:19.979 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 10 librados: watch_flush enter 2023-09-30T00:51:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 10 librados: watch_flush exit 2023-09-30T00:51:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 20 client.4696.objecter shutdown clearing up homeless session... 2023-09-30T00:51:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 10 client.4696.objecter successfully canceled tick 2023-09-30T00:51:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/407690790 >> v2:172.21.15.135:6800/116426 conn(0x7f7aec068890 msgr2=0x7f7aec06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:19.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 1 --2- 172.21.15.40:0/407690790 >> v2:172.21.15.135:6800/116426 conn(0x7f7aec068890 0x7f7aec06ad40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7af8007fd0 tx=0x7f7af8009040 comp rx=0 tx=0).stop 2023-09-30T00:51:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 10 monclient: shutdown 2023-09-30T00:51:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/407690790 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b080b1460 msgr2=0x7f7b0813e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 1 --2- 172.21.15.40:0/407690790 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b080b1460 0x7f7b0813e1f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400c490 tx=0x7f7b04007ea0 comp rx=0 tx=0).stop 2023-09-30T00:51:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/407690790 shutdown_connections 2023-09-30T00:51:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 1 --2- 172.21.15.40:0/407690790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7b080b1830 0x7f7b080a9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 1 --2- 172.21.15.40:0/407690790 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7b080a5fb0 0x7f7b080a8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 1 --2- 172.21.15.40:0/407690790 >> v2:172.21.15.135:6800/116426 conn(0x7f7aec068890 0x7f7aec06ad40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 1 --2- 172.21.15.40:0/407690790 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7b080b1460 0x7f7b0813e1f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/407690790 >> 172.21.15.40:0/407690790 conn(0x7f7b080631f0 msgr2=0x7f7b080b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:19.982 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/407690790 shutdown_connections 2023-09-30T00:51:19.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 1 -- 172.21.15.40:0/407690790 wait complete. 2023-09-30T00:51:19.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:19.980+0000 7f7b0fe9a700 1 librados: shutdown 2023-09-30T00:51:22.996 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:23.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.133+0000 7f96b1870700 10 monclient: get_monmap_and_config 2023-09-30T00:51:23.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.133+0000 7f96b1870700 10 monclient: build_initial_monmap 2023-09-30T00:51:23.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.133+0000 7f96b1870700 10 monclient: monmap: 2023-09-30T00:51:23.134 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:23.134 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:23.135 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:23.134921+0000 2023-09-30T00:51:23.135 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:23.134921+0000 2023-09-30T00:51:23.135 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:23.135 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:23.135 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:23.135 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:23.135 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:23.135 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:23.135 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:23.135 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:23.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.134+0000 7f96b1870700 1 Processor -- start 2023-09-30T00:51:23.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.134+0000 7f96b1870700 1 -- start start 2023-09-30T00:51:23.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.134+0000 7f96b1870700 10 monclient: init 2023-09-30T00:51:23.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96b1870700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:23.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96b1870700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-09-30T00:51:23.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96b1870700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96ac0a5710 0x7f96ac0a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:23.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96b1870700 10 monclient(hunting): picked mon.noname-b con 0x7f96ac0a5710 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:23.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96b1870700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f96ac0a7b20 0x7f96ac13e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:23.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96b1870700 10 monclient(hunting): picked mon.noname-c con 0x7f96ac0a7b20 addr v2:172.21.15.135:3300/0 2023-09-30T00:51:23.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96b1870700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96ac13e710 addr v1:172.21.15.40:6789/0 2023-09-30T00:51:23.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96b1870700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96ac0a5ae0 con 0x7f96ac13e710 2023-09-30T00:51:23.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96b1870700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:23.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96b1870700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96ac0a7ff0 con 0x7f96ac0a7b20 2023-09-30T00:51:23.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96b1870700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:23.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96b1870700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac0aedc0 con 0x7f96ac0a5710 2023-09-30T00:51:23.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96b1870700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:23.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96b1870700 10 monclient(hunting): authenticate will time out at 5544.436249s 2023-09-30T00:51:23.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96ab7fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f96ac13e710 0x7f96ac141760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55228/0 (socket says 172.21.15.40:55228) 2023-09-30T00:51:23.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96ab7fe700 1 -- 172.21.15.40:0/2217497231 learned_addr learned my addr 172.21.15.40:0/2217497231 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:51:23.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.135+0000 7f96aaffd700 1 --2- 172.21.15.40:0/2217497231 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96ac0a5710 0x7f96ac0a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:23.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.136+0000 7f96aa7fc700 1 --2- 172.21.15.40:0/2217497231 >> v2:172.21.15.135:3300/0 conn(0x7f96ac0a7b20 0x7f96ac13e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:23.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.136+0000 7f96aaffd700 10 monclient(hunting): get_auth_request con 0x7f96ac0a5710 auth_method 0 2023-09-30T00:51:23.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.136+0000 7f96aaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:23.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.136+0000 7f96aaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:23.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.136+0000 7f96aaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:23.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.136+0000 7f96aa7fc700 10 monclient(hunting): get_auth_request con 0x7f96ac0a7b20 auth_method 0 2023-09-30T00:51:23.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.136+0000 7f96aa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:23.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.136+0000 7f96aa7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:23.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.136+0000 7f96aa7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:23.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.136+0000 7f96aaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:23.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.137+0000 7f96aaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:23.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.137+0000 7f96aaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:23.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.137+0000 7f96aa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:23.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.137+0000 7f96aa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:23.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.137+0000 7f96aa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.138+0000 7f96a9ffb700 1 -- 172.21.15.40:0/2217497231 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199642303 0 0) 0x7f96ac0a5ae0 con 0x7f96ac13e710 2023-09-30T00:51:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.138+0000 7f96a9ffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.138+0000 7f96a9ffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.138+0000 7f96a9ffb700 10 monclient(hunting): my global_id is 4632 2023-09-30T00:51:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.138+0000 7f96a9ffb700 1 -- 172.21.15.40:0/2217497231 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9698003570 con 0x7f96ac13e710 2023-09-30T00:51:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.139+0000 7f96aaffd700 10 monclient(hunting): handle_auth_done global_id 4562 payload 274 2023-09-30T00:51:23.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.139+0000 7f96aaffd700 1 -- 172.21.15.40:0/2217497231 >> v2:172.21.15.135:3300/0 conn(0x7f96ac0a7b20 msgr2=0x7f96ac13e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.139+0000 7f96aaffd700 1 --2- 172.21.15.40:0/2217497231 >> v2:172.21.15.135:3300/0 conn(0x7f96ac0a7b20 0x7f96ac13e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.139+0000 7f96aaffd700 1 -- 172.21.15.40:0/2217497231 >> v1:172.21.15.40:6789/0 conn(0x7f96ac13e710 legacy=0x7f96ac141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.139+0000 7f96aaffd700 10 monclient: _finish_hunting 0 2023-09-30T00:51:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.139+0000 7f96aaffd700 1 monclient: found mon.noname-b 2023-09-30T00:51:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.139+0000 7f96aaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.139+0000 7f96aaffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:51:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.139+0000 7f96aaffd700 1 -- 172.21.15.40:0/2217497231 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac0aef00 con 0x7f96ac0a5710 2023-09-30T00:51:23.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.142+0000 7f96aaffd700 1 --2- 172.21.15.40:0/2217497231 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96ac0a5710 0x7f96ac0a75e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f969c0098e0 tx=0x7f969c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bae52a434b07c49 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:23.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96a9ffb700 1 -- 172.21.15.40:0/2217497231 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c011630 con 0x7f96ac0a5710 2023-09-30T00:51:23.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:23.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96a9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:23.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96a9ffb700 10 monclient: dump: 2023-09-30T00:51:23.147 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:23.147 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:23.147 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:23.147 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:23.148 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96a9ffb700 10 monclient: _finish_auth 0 2023-09-30T00:51:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96a9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96a9ffb700 1 -- 172.21.15.40:0/2217497231 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969c011790 con 0x7f96ac0a5710 2023-09-30T00:51:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96b1870700 5 monclient: authenticate success, global_id 4562 2023-09-30T00:51:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96b1870700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:51:23.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96a9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:23.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96b1870700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:23.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96a9ffb700 1 -- 172.21.15.40:0/2217497231 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c011b30 con 0x7f96ac0a5710 2023-09-30T00:51:23.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96b1870700 4 set_mon_vals no callback set 2023-09-30T00:51:23.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:23.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96a9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:51:23.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.146+0000 7f96a9ffb700 10 monclient: dump: 2023-09-30T00:51:23.149 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:23.149 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:23.149 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:23.149 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:23.149 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:23.149 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:23.149 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:23.150 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:23.150 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:23.150 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.147+0000 7f96b1870700 10 monclient: shutdown 2023-09-30T00:51:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.147+0000 7f96b1870700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.147+0000 7f96b1870700 1 -- 172.21.15.40:0/2217497231 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96ac0a5710 msgr2=0x7f96ac0a75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.147+0000 7f96b1870700 1 --2- 172.21.15.40:0/2217497231 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96ac0a5710 0x7f96ac0a75e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f969c0098e0 tx=0x7f969c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:51:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.147+0000 7f96b1870700 1 -- 172.21.15.40:0/2217497231 shutdown_connections 2023-09-30T00:51:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.147+0000 7f96b1870700 1 --2- 172.21.15.40:0/2217497231 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96ac0a5710 0x7f96ac0a75e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.147+0000 7f96b1870700 1 --2- 172.21.15.40:0/2217497231 >> v2:172.21.15.135:3300/0 conn(0x7f96ac0a7b20 0x7f96ac13e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.147+0000 7f96b1870700 1 -- 172.21.15.40:0/2217497231 >> 172.21.15.40:0/2217497231 conn(0x7f96ac069080 msgr2=0x7f96ac0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.147+0000 7f96b1870700 1 -- 172.21.15.40:0/2217497231 shutdown_connections 2023-09-30T00:51:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.147+0000 7f96b1870700 1 -- 172.21.15.40:0/2217497231 wait complete. 2023-09-30T00:51:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.148+0000 7f96b1870700 10 monclient: build_initial_monmap 2023-09-30T00:51:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.148+0000 7f96b1870700 10 monclient: monmap: 2023-09-30T00:51:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:23.154 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:23.155 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.149+0000 7f96b1870700 1 librados: starting msgr at 2023-09-30T00:51:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.149+0000 7f96b1870700 1 librados: starting objecter 2023-09-30T00:51:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.149+0000 7f96b1870700 1 Processor -- start 2023-09-30T00:51:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.149+0000 7f96b1870700 1 -- start start 2023-09-30T00:51:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.149+0000 7f96b1870700 1 librados: setting wanted keys 2023-09-30T00:51:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.149+0000 7f96b1870700 1 librados: calling monclient init 2023-09-30T00:51:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.149+0000 7f96b1870700 10 monclient: init 2023-09-30T00:51:23.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:51:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96ac0a5710 0x7f96ac13acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 10 monclient(hunting): picked mon.noname-a con 0x7f96ac0a5710 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96ac0a7b20 0x7f96ac13d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 10 monclient(hunting): picked mon.noname-b con 0x7f96ac0a7b20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96ac13e710 0x7f96ac14d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 10 monclient(hunting): picked mon.noname-c con 0x7f96ac13e710 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac13b450 con 0x7f96ac0a5710 2023-09-30T00:51:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac13aa40 con 0x7f96ac0a7b20 2023-09-30T00:51:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac13da60 con 0x7f96ac13e710 2023-09-30T00:51:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96b1870700 10 monclient(hunting): authenticate will time out at 5544.451270s 2023-09-30T00:51:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96ab7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96ac13e710 0x7f96ac14d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.150+0000 7f96aaffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96ac0a5710 0x7f96ac13acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96ab7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96ac13e710 0x7f96ac14d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:40102/0 (socket says 172.21.15.40:40102) 2023-09-30T00:51:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96ab7fe700 1 -- 172.21.15.40:0/826116523 learned_addr learned my addr 172.21.15.40:0/826116523 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96ab7fe700 10 monclient(hunting): get_auth_request con 0x7f96ac13e710 auth_method 0 2023-09-30T00:51:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96ab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96ab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:23.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96aa7fc700 1 --2- 172.21.15.40:0/826116523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96ac0a7b20 0x7f96ac13d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:23.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96aaffd700 10 monclient(hunting): get_auth_request con 0x7f96ac0a5710 auth_method 0 2023-09-30T00:51:23.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96aaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:23.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96aaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:23.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96aaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:23.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96aa7fc700 10 monclient(hunting): get_auth_request con 0x7f96ac0a7b20 auth_method 0 2023-09-30T00:51:23.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96aa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:23.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96aa7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:23.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96aa7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:23.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:23.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:23.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.151+0000 7f96ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.152+0000 7f96aa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.152+0000 7f96aa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.152+0000 7f96aa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.152+0000 7f96aaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.152+0000 7f96aaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:23.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.152+0000 7f96aaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.155+0000 7f96ab7fe700 10 monclient(hunting): handle_auth_done global_id 4565 payload 824 2023-09-30T00:51:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.155+0000 7f96ab7fe700 1 -- 172.21.15.40:0/826116523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96ac0a7b20 msgr2=0x7f96ac13d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.155+0000 7f96ab7fe700 1 --2- 172.21.15.40:0/826116523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96ac0a7b20 0x7f96ac13d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.155+0000 7f96ab7fe700 1 -- 172.21.15.40:0/826116523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96ac0a5710 msgr2=0x7f96ac13acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.155+0000 7f96ab7fe700 1 --2- 172.21.15.40:0/826116523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96ac0a5710 0x7f96ac13acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.155+0000 7f96ab7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:51:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.155+0000 7f96ab7fe700 1 monclient: found mon.noname-c 2023-09-30T00:51:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.155+0000 7f96ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:23.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.155+0000 7f96ab7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:51:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.155+0000 7f96ab7fe700 1 -- 172.21.15.40:0/826116523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c009420 con 0x7f96ac13e710 2023-09-30T00:51:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.157+0000 7f96ab7fe700 1 --2- 172.21.15.40:0/826116523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96ac13e710 0x7f96ac14d9b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f96a4001460 tx=0x7f96a4001a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.157+0000 7f96a2ffd700 10 client.?.objecter ms_handle_connect 0x7f96ac13e710 2023-09-30T00:51:23.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.157+0000 7f96a2ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96a2ffd700 1 -- 172.21.15.40:0/826116523 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a400a870 con 0x7f96ac13e710 2023-09-30T00:51:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96a2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:51:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96a2ffd700 10 monclient: dump: 2023-09-30T00:51:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:23.163 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:23.164 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96a2ffd700 10 monclient: _finish_auth 0 2023-09-30T00:51:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96a2ffd700 1 -- 172.21.15.40:0/826116523 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a400a9d0 con 0x7f96ac13e710 2023-09-30T00:51:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96b1870700 5 monclient: authenticate success, global_id 4565 2023-09-30T00:51:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96b1870700 10 monclient: _renew_subs 2023-09-30T00:51:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96b1870700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96b1870700 1 -- 172.21.15.40:0/826116523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac13e360 con 0x7f96ac13e710 2023-09-30T00:51:23.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96a2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96a2ffd700 1 -- 172.21.15.40:0/826116523 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a400ad50 con 0x7f96ac13e710 2023-09-30T00:51:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96a2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:51:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96a97fa700 4 set_mon_vals no callback set 2023-09-30T00:51:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96a2ffd700 10 monclient: dump: 2023-09-30T00:51:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:23.165 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:23.166 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96b1870700 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96b1870700 10 monclient: _renew_subs 2023-09-30T00:51:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96b1870700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96b1870700 1 -- 172.21.15.40:0/826116523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac14def0 con 0x7f96ac13e710 2023-09-30T00:51:23.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.162+0000 7f96b1870700 1 librados: init done 2023-09-30T00:51:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.163+0000 7f96b1870700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.163+0000 7f96b1870700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.163+0000 7f96b1870700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:23.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.163+0000 7f96b1870700 1 -- 172.21.15.40:0/826116523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac072b90 con 0x7f96ac13e710 2023-09-30T00:51:23.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.423+0000 7f96a2ffd700 1 -- 172.21.15.40:0/826116523 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f96a4007300 con 0x7f96ac13e710 2023-09-30T00:51:23.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.423+0000 7f96a2ffd700 1 --2- 172.21.15.40:0/826116523 >> v2:172.21.15.135:6800/116426 conn(0x7f968c068890 0x7f968c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:23.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.424+0000 7f96aaffd700 1 --2- 172.21.15.40:0/826116523 >> v2:172.21.15.135:6800/116426 conn(0x7f968c068890 0x7f968c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:23.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.424+0000 7f96aaffd700 10 monclient: get_auth_request con 0x7f968c068890 auth_method 0 2023-09-30T00:51:23.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.424+0000 7f96aaffd700 1 --2- 172.21.15.40:0/826116523 >> v2:172.21.15.135:6800/116426 conn(0x7f968c068890 0x7f968c06ad40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f969c0091e0 tx=0x7f969c017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:23.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.424+0000 7f96a2ffd700 10 client.4565.objecter ms_handle_connect 0x7f968c068890 2023-09-30T00:51:23.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.429+0000 7f96a2ffd700 1 -- 172.21.15.40:0/826116523 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f96a4056990 con 0x7f96ac13e710 2023-09-30T00:51:23.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.429+0000 7f96a2ffd700 10 client.4565.objecter ms_dispatch 0x7f96ac001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:23.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.429+0000 7f96a2ffd700 3 client.4565.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:23.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.429+0000 7f96a2ffd700 3 client.4565.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:23.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.429+0000 7f96a2ffd700 20 client.4565.objecter dump_active .. 0 homeless 2023-09-30T00:51:23.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.459+0000 7f96a2ffd700 1 -- 172.21.15.40:0/826116523 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f96a40525d0 con 0x7f96ac13e710 2023-09-30T00:51:23.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.459+0000 7f96a2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:23.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.459+0000 7f96a2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:23.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.719+0000 7f96b1870700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:23.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.719+0000 7f96b1870700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:23.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.719+0000 7f96b1870700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:23.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.719+0000 7f96b1870700 1 -- 172.21.15.40:0/826116523 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f96ac13ecb0 con 0x7f96ac13e710 2023-09-30T00:51:23.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.737+0000 7f96a2ffd700 1 -- 172.21.15.40:0/826116523 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f96a4055bf0 con 0x7f96ac13e710 2023-09-30T00:51:23.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.737+0000 7f96a2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:23.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.737+0000 7f96a2ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:23.738 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:23.738 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:23.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 10 librados: watch_flush enter 2023-09-30T00:51:23.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 10 librados: watch_flush exit 2023-09-30T00:51:23.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 20 client.4565.objecter shutdown clearing up homeless session... 2023-09-30T00:51:23.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 10 client.4565.objecter successfully canceled tick 2023-09-30T00:51:23.740 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 1 -- 172.21.15.40:0/826116523 >> v2:172.21.15.135:6800/116426 conn(0x7f968c068890 msgr2=0x7f968c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 1 --2- 172.21.15.40:0/826116523 >> v2:172.21.15.135:6800/116426 conn(0x7f968c068890 0x7f968c06ad40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f969c0091e0 tx=0x7f969c017000 comp rx=0 tx=0).stop 2023-09-30T00:51:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 10 monclient: shutdown 2023-09-30T00:51:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 1 -- 172.21.15.40:0/826116523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96ac13e710 msgr2=0x7f96ac14d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 1 --2- 172.21.15.40:0/826116523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96ac13e710 0x7f96ac14d9b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f96a4001460 tx=0x7f96a4001a40 comp rx=0 tx=0).stop 2023-09-30T00:51:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 1 -- 172.21.15.40:0/826116523 shutdown_connections 2023-09-30T00:51:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 1 --2- 172.21.15.40:0/826116523 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96ac13e710 0x7f96ac14d9b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 1 --2- 172.21.15.40:0/826116523 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96ac0a5710 0x7f96ac13acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 1 --2- 172.21.15.40:0/826116523 >> v2:172.21.15.135:6800/116426 conn(0x7f968c068890 0x7f968c06ad40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 1 --2- 172.21.15.40:0/826116523 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f96ac0a7b20 0x7f96ac13d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:23.741 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 1 -- 172.21.15.40:0/826116523 >> 172.21.15.40:0/826116523 conn(0x7f96ac069080 msgr2=0x7f96ac0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:23.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 1 -- 172.21.15.40:0/826116523 shutdown_connections 2023-09-30T00:51:23.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 1 -- 172.21.15.40:0/826116523 wait complete. 2023-09-30T00:51:23.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:23.739+0000 7f96b1870700 1 librados: shutdown 2023-09-30T00:51:26.755 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.895+0000 7fef6bffd700 10 monclient: get_monmap_and_config 2023-09-30T00:51:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.895+0000 7fef6bffd700 10 monclient: build_initial_monmap 2023-09-30T00:51:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.895+0000 7fef6bffd700 10 monclient: monmap: 2023-09-30T00:51:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:26.896182+0000 2023-09-30T00:51:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:26.896182+0000 2023-09-30T00:51:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:26.897 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.895+0000 7fef6bffd700 1 Processor -- start 2023-09-30T00:51:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 1 -- start start 2023-09-30T00:51:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 10 monclient: init 2023-09-30T00:51:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-09-30T00:51:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fef640b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:51:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fef640b1830 0x7fef640a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 10 monclient(hunting): picked mon.noname-c con 0x7fef640b1830 addr v2:172.21.15.135:3300/0 2023-09-30T00:51:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef6413ac20 0x7fef6413b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 10 monclient(hunting): picked mon.noname-b con 0x7fef6413ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef640a5b20 con 0x7fef640b1460 2023-09-30T00:51:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef64141e40 con 0x7fef640b1830 2023-09-30T00:51:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef640a64f0 con 0x7fef6413ac20 2023-09-30T00:51:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.896+0000 7fef6bffd700 10 monclient(hunting): authenticate will time out at 5548.197828s 2023-09-30T00:51:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.897+0000 7fef69d99700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fef640b1460 0x7fef64141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55244/0 (socket says 172.21.15.40:55244) 2023-09-30T00:51:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.897+0000 7fef69d99700 1 -- 172.21.15.40:0/1263001314 learned_addr learned my addr 172.21.15.40:0/1263001314 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:51:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.897+0000 7fef6a59a700 1 --2- 172.21.15.40:0/1263001314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef6413ac20 0x7fef6413b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.897+0000 7fef69598700 1 --2- 172.21.15.40:0/1263001314 >> v2:172.21.15.135:3300/0 conn(0x7fef640b1830 0x7fef640a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.897+0000 7fef6a59a700 10 monclient(hunting): get_auth_request con 0x7fef6413ac20 auth_method 0 2023-09-30T00:51:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.897+0000 7fef6a59a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:26.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.897+0000 7fef6a59a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.897+0000 7fef6a59a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.897+0000 7fef69598700 10 monclient(hunting): get_auth_request con 0x7fef640b1830 auth_method 0 2023-09-30T00:51:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.898+0000 7fef69598700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.898+0000 7fef69598700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.898+0000 7fef69598700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.898+0000 7fef69598700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.898+0000 7fef69598700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.898+0000 7fef69598700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.898+0000 7fef6a59a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.898+0000 7fef6a59a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.898+0000 7fef6a59a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:26.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.901+0000 7fef69598700 10 monclient(hunting): handle_auth_done global_id 4705 payload 274 2023-09-30T00:51:26.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.901+0000 7fef69598700 1 -- 172.21.15.40:0/1263001314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef6413ac20 msgr2=0x7fef6413b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:26.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.901+0000 7fef69598700 1 --2- 172.21.15.40:0/1263001314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef6413ac20 0x7fef6413b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:26.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.901+0000 7fef69598700 1 -- 172.21.15.40:0/1263001314 >> v1:172.21.15.40:6789/0 conn(0x7fef640b1460 legacy=0x7fef64141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.901+0000 7fef69598700 10 monclient: _finish_hunting 0 2023-09-30T00:51:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.901+0000 7fef69598700 1 monclient: found mon.noname-c 2023-09-30T00:51:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.901+0000 7fef69598700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.901+0000 7fef69598700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:51:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.901+0000 7fef69598700 1 -- 172.21.15.40:0/1263001314 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef640a71f0 con 0x7fef640b1830 2023-09-30T00:51:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.901+0000 7fef68d97700 1 -- 172.21.15.40:0/1263001314 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855317365 0 0) 0x7fef640a5b20 con 0x7fef640b1460 2023-09-30T00:51:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.901+0000 7fef68d97700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T00:51:26.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.903+0000 7fef69598700 1 --2- 172.21.15.40:0/1263001314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef640b1830 0x7fef640a5fb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fef54009910 tx=0x7fef540091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6d6e7bb72b74b26 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.907+0000 7fef68d97700 1 -- 172.21.15.40:0/1263001314 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5400b640 con 0x7fef640b1830 2023-09-30T00:51:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.907+0000 7fef68d97700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.907+0000 7fef68d97700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:51:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.907+0000 7fef68d97700 10 monclient: dump: 2023-09-30T00:51:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:26.908 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:26.909 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.907+0000 7fef68d97700 10 monclient: _finish_auth 0 2023-09-30T00:51:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.907+0000 7fef68d97700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:26.909 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.907+0000 7fef68d97700 1 -- 172.21.15.40:0/1263001314 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5400b7a0 con 0x7fef640b1830 2023-09-30T00:51:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.907+0000 7fef68d97700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.907+0000 7fef68d97700 1 -- 172.21.15.40:0/1263001314 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5400bb60 con 0x7fef640b1830 2023-09-30T00:51:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.907+0000 7fef68d97700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.907+0000 7fef68d97700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.907+0000 7fef68d97700 10 monclient: dump: 2023-09-30T00:51:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:26.910 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:26.911 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.908+0000 7fef6bffd700 5 monclient: authenticate success, global_id 4705 2023-09-30T00:51:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.908+0000 7fef6bffd700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:26.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.908+0000 7fef6bffd700 4 set_mon_vals no callback set 2023-09-30T00:51:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.908+0000 7fef6bffd700 10 monclient: shutdown 2023-09-30T00:51:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.908+0000 7fef6bffd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.908+0000 7fef6bffd700 1 -- 172.21.15.40:0/1263001314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef640b1830 msgr2=0x7fef640a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:26.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.908+0000 7fef6bffd700 1 --2- 172.21.15.40:0/1263001314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef640b1830 0x7fef640a5fb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fef54009910 tx=0x7fef540091a0 comp rx=0 tx=0).stop 2023-09-30T00:51:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.908+0000 7fef6bffd700 1 -- 172.21.15.40:0/1263001314 shutdown_connections 2023-09-30T00:51:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.908+0000 7fef6bffd700 1 --2- 172.21.15.40:0/1263001314 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef6413ac20 0x7fef6413b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.908+0000 7fef6bffd700 1 --2- 172.21.15.40:0/1263001314 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef640b1830 0x7fef640a5fb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.908+0000 7fef6bffd700 1 -- 172.21.15.40:0/1263001314 >> 172.21.15.40:0/1263001314 conn(0x7fef640631f0 msgr2=0x7fef640635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.908+0000 7fef6bffd700 1 -- 172.21.15.40:0/1263001314 shutdown_connections 2023-09-30T00:51:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.908+0000 7fef6bffd700 1 -- 172.21.15.40:0/1263001314 wait complete. 2023-09-30T00:51:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.909+0000 7fef6bffd700 10 monclient: build_initial_monmap 2023-09-30T00:51:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.909+0000 7fef6bffd700 10 monclient: monmap: 2023-09-30T00:51:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:26.915 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:26.916 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.909+0000 7fef6bffd700 1 librados: starting msgr at 2023-09-30T00:51:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.909+0000 7fef6bffd700 1 librados: starting objecter 2023-09-30T00:51:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.909+0000 7fef6bffd700 1 Processor -- start 2023-09-30T00:51:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 1 -- start start 2023-09-30T00:51:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 1 librados: setting wanted keys 2023-09-30T00:51:26.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 1 librados: calling monclient init 2023-09-30T00:51:26.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 10 monclient: init 2023-09-30T00:51:26.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:26.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:51:26.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef640b1460 0x7fef6413ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:26.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 10 monclient(hunting): picked mon.noname-b con 0x7fef640b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:26.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef640b1830 0x7fef640a93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:26.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 10 monclient(hunting): picked mon.noname-c con 0x7fef640b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:26.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef6413ac20 0x7fef6414d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:26.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 10 monclient(hunting): picked mon.noname-a con 0x7fef6413ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:26.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:26.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef640a9cb0 con 0x7fef6413ac20 2023-09-30T00:51:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef640a98e0 con 0x7fef640b1460 2023-09-30T00:51:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6413e9b0 con 0x7fef640b1830 2023-09-30T00:51:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.910+0000 7fef6bffd700 10 monclient(hunting): authenticate will time out at 5548.211630s 2023-09-30T00:51:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.911+0000 7fef69598700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef640b1830 0x7fef640a93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:26.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.911+0000 7fef6a59a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef6413ac20 0x7fef6414d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:26.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.911+0000 7fef69598700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef640b1830 0x7fef640a93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:40118/0 (socket says 172.21.15.40:40118) 2023-09-30T00:51:26.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.911+0000 7fef69598700 1 -- 172.21.15.40:0/1140108448 learned_addr learned my addr 172.21.15.40:0/1140108448 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:26.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.911+0000 7fef69598700 10 monclient(hunting): get_auth_request con 0x7fef640b1830 auth_method 0 2023-09-30T00:51:26.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.911+0000 7fef69598700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:26.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.911+0000 7fef69598700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:26.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.911+0000 7fef69598700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:26.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.911+0000 7fef6a59a700 10 monclient(hunting): get_auth_request con 0x7fef6413ac20 auth_method 0 2023-09-30T00:51:26.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.911+0000 7fef6a59a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:26.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.911+0000 7fef6a59a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:26.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.911+0000 7fef6a59a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:26.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.912+0000 7fef69598700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:26.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.912+0000 7fef69598700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.912+0000 7fef69598700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.912+0000 7fef69d99700 1 --2- 172.21.15.40:0/1140108448 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef640b1460 0x7fef6413ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.912+0000 7fef6a59a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.912+0000 7fef6a59a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.912+0000 7fef6a59a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.912+0000 7fef69d99700 10 monclient(hunting): get_auth_request con 0x7fef640b1460 auth_method 0 2023-09-30T00:51:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.912+0000 7fef69d99700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.912+0000 7fef69d99700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.912+0000 7fef69d99700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.913+0000 7fef69d99700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:26.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.913+0000 7fef69d99700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:26.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.913+0000 7fef69d99700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.915+0000 7fef69598700 10 monclient(hunting): handle_auth_done global_id 4571 payload 824 2023-09-30T00:51:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.915+0000 7fef69598700 1 -- 172.21.15.40:0/1140108448 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef640b1460 msgr2=0x7fef6413ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.915+0000 7fef69598700 1 --2- 172.21.15.40:0/1140108448 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef640b1460 0x7fef6413ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.915+0000 7fef69598700 1 -- 172.21.15.40:0/1140108448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef6413ac20 msgr2=0x7fef6414d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.915+0000 7fef69598700 1 --2- 172.21.15.40:0/1140108448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef6413ac20 0x7fef6414d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.915+0000 7fef69598700 10 monclient: _finish_hunting 0 2023-09-30T00:51:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.915+0000 7fef69598700 1 monclient: found mon.noname-c 2023-09-30T00:51:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.915+0000 7fef69598700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:26.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.915+0000 7fef69598700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:51:26.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.915+0000 7fef69598700 1 -- 172.21.15.40:0/1140108448 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef54009420 con 0x7fef640b1830 2023-09-30T00:51:26.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.918+0000 7fef69598700 1 --2- 172.21.15.40:0/1140108448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef640b1830 0x7fef640a93a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef6406f5e0 tx=0x7fef54017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:26.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.918+0000 7fef59ffb700 10 client.?.objecter ms_handle_connect 0x7fef640b1830 2023-09-30T00:51:26.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.918+0000 7fef59ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:26.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef59ffb700 1 -- 172.21.15.40:0/1140108448 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5401f070 con 0x7fef640b1830 2023-09-30T00:51:26.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:26.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef59ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:51:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef59ffb700 10 monclient: dump: 2023-09-30T00:51:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:26.925 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef59ffb700 10 monclient: _finish_auth 0 2023-09-30T00:51:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef59ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef6bffd700 5 monclient: authenticate success, global_id 4571 2023-09-30T00:51:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef6bffd700 10 monclient: _renew_subs 2023-09-30T00:51:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef6bffd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef6bffd700 1 -- 172.21.15.40:0/1140108448 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef640aa140 con 0x7fef640b1830 2023-09-30T00:51:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef6bffd700 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef6bffd700 10 monclient: _renew_subs 2023-09-30T00:51:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef6bffd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef6bffd700 1 -- 172.21.15.40:0/1140108448 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef6414dde0 con 0x7fef640b1830 2023-09-30T00:51:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.921+0000 7fef6bffd700 1 librados: init done 2023-09-30T00:51:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.922+0000 7fef6bffd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.922+0000 7fef6bffd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.922+0000 7fef6bffd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:26.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.922+0000 7fef6bffd700 1 -- 172.21.15.40:0/1140108448 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef640a9a50 con 0x7fef640b1830 2023-09-30T00:51:26.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.922+0000 7fef59ffb700 1 -- 172.21.15.40:0/1140108448 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef54002f30 con 0x7fef640b1830 2023-09-30T00:51:26.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.922+0000 7fef59ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:26.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.922+0000 7fef59ffb700 1 -- 172.21.15.40:0/1140108448 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef54006bc0 con 0x7fef640b1830 2023-09-30T00:51:26.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.922+0000 7fef59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:26.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.922+0000 7fef59ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:51:26.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.922+0000 7fef59ffb700 10 monclient: dump: 2023-09-30T00:51:26.929 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:26.929 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:26.930 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:26.930 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:26.930 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:26.930 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:26.930 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:26.930 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:26.930 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:26.930 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:26.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:26.923+0000 7fef5bfff700 4 set_mon_vals no callback set 2023-09-30T00:51:27.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.226+0000 7fef59ffb700 1 -- 172.21.15.40:0/1140108448 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fef54023070 con 0x7fef640b1830 2023-09-30T00:51:27.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.226+0000 7fef59ffb700 1 --2- 172.21.15.40:0/1140108448 >> v2:172.21.15.135:6800/116426 conn(0x7fef48068890 0x7fef4806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:27.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.226+0000 7fef59ffb700 1 -- 172.21.15.40:0/1140108448 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fef540856e0 con 0x7fef640b1830 2023-09-30T00:51:27.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.226+0000 7fef59ffb700 10 client.4571.objecter ms_dispatch 0x7fef64001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:27.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.226+0000 7fef59ffb700 3 client.4571.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:27.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.226+0000 7fef59ffb700 3 client.4571.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:27.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.226+0000 7fef59ffb700 20 client.4571.objecter dump_active .. 0 homeless 2023-09-30T00:51:27.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.226+0000 7fef59ffb700 1 -- 172.21.15.40:0/1140108448 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fef54085b30 con 0x7fef640b1830 2023-09-30T00:51:27.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.226+0000 7fef59ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:27.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.226+0000 7fef59ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:27.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.226+0000 7fef69d99700 1 --2- 172.21.15.40:0/1140108448 >> v2:172.21.15.135:6800/116426 conn(0x7fef48068890 0x7fef4806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:27.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.226+0000 7fef69d99700 10 monclient: get_auth_request con 0x7fef48068890 auth_method 0 2023-09-30T00:51:27.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.227+0000 7fef59ffb700 10 client.4571.objecter ms_handle_connect 0x7fef48068890 2023-09-30T00:51:27.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.227+0000 7fef69d99700 1 --2- 172.21.15.40:0/1140108448 >> v2:172.21.15.135:6800/116426 conn(0x7fef48068890 0x7fef4806ad40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fef60007160 tx=0x7fef60007f50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:27.492 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.491+0000 7fef6bffd700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:27.492 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.491+0000 7fef6bffd700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:27.492 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.491+0000 7fef6bffd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:27.492 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.491+0000 7fef6bffd700 1 -- 172.21.15.40:0/1140108448 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fef64064a00 con 0x7fef640b1830 2023-09-30T00:51:27.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.496+0000 7fef59ffb700 1 -- 172.21.15.40:0/1140108448 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fef54053030 con 0x7fef640b1830 2023-09-30T00:51:27.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.496+0000 7fef59ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:27.497 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.496+0000 7fef59ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:27.497 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:27.497 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 10 librados: watch_flush enter 2023-09-30T00:51:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 10 librados: watch_flush exit 2023-09-30T00:51:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 20 client.4571.objecter shutdown clearing up homeless session... 2023-09-30T00:51:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 10 client.4571.objecter successfully canceled tick 2023-09-30T00:51:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 1 -- 172.21.15.40:0/1140108448 >> v2:172.21.15.135:6800/116426 conn(0x7fef48068890 msgr2=0x7fef4806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 1 --2- 172.21.15.40:0/1140108448 >> v2:172.21.15.135:6800/116426 conn(0x7fef48068890 0x7fef4806ad40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fef60007160 tx=0x7fef60007f50 comp rx=0 tx=0).stop 2023-09-30T00:51:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 10 monclient: shutdown 2023-09-30T00:51:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 1 -- 172.21.15.40:0/1140108448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef640b1830 msgr2=0x7fef640a93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 1 --2- 172.21.15.40:0/1140108448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef640b1830 0x7fef640a93a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef6406f5e0 tx=0x7fef54017000 comp rx=0 tx=0).stop 2023-09-30T00:51:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 1 -- 172.21.15.40:0/1140108448 shutdown_connections 2023-09-30T00:51:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 1 --2- 172.21.15.40:0/1140108448 >> v2:172.21.15.135:6800/116426 conn(0x7fef48068890 0x7fef4806ad40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 1 --2- 172.21.15.40:0/1140108448 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fef640b1460 0x7fef6413ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 1 --2- 172.21.15.40:0/1140108448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef6413ac20 0x7fef6414d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 1 --2- 172.21.15.40:0/1140108448 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fef640b1830 0x7fef640a93a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 1 -- 172.21.15.40:0/1140108448 >> 172.21.15.40:0/1140108448 conn(0x7fef640631f0 msgr2=0x7fef640b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:27.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 1 -- 172.21.15.40:0/1140108448 shutdown_connections 2023-09-30T00:51:27.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 1 -- 172.21.15.40:0/1140108448 wait complete. 2023-09-30T00:51:27.502 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:27.499+0000 7fef6bffd700 1 librados: shutdown 2023-09-30T00:51:30.515 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.649+0000 7fec0896f700 10 monclient: get_monmap_and_config 2023-09-30T00:51:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.649+0000 7fec0896f700 10 monclient: build_initial_monmap 2023-09-30T00:51:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.649+0000 7fec0896f700 10 monclient: monmap: 2023-09-30T00:51:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:30.650516+0000 2023-09-30T00:51:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:30.650516+0000 2023-09-30T00:51:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:30.651 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.650+0000 7fec0896f700 1 Processor -- start 2023-09-30T00:51:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.650+0000 7fec0896f700 1 -- start start 2023-09-30T00:51:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.650+0000 7fec0896f700 10 monclient: init 2023-09-30T00:51:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.651+0000 7fec0896f700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.651+0000 7fec0896f700 10 monclient: _add_conns ranks=[2,1,3,4,0] 2023-09-30T00:51:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.651+0000 7fec0896f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec040b1460 0x7fec040b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.651+0000 7fec0896f700 10 monclient(hunting): picked mon.noname-b con 0x7fec040b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.651+0000 7fec0896f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec040a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:51:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.651+0000 7fec0896f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fec0413ac20 0x7fec0413b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.651+0000 7fec0896f700 10 monclient(hunting): picked mon.noname-a con 0x7fec0413ac20 addr v2:172.21.15.40:3300/0 2023-09-30T00:51:30.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.651+0000 7fec0896f700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec040a5b20 con 0x7fec040a5fb0 2023-09-30T00:51:30.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.651+0000 7fec0896f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:30.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.651+0000 7fec0896f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec040a71f0 con 0x7fec0413ac20 2023-09-30T00:51:30.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.651+0000 7fec0896f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:30.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.651+0000 7fec0896f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec040a74d0 con 0x7fec040b1460 2023-09-30T00:51:30.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.651+0000 7fec0896f700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:30.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.651+0000 7fec0896f700 10 monclient(hunting): authenticate will time out at 5551.952182s 2023-09-30T00:51:30.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.652+0000 7fec02d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fec0413ac20 0x7fec0413b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:30.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.654+0000 7fec02d9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fec0413ac20 0x7fec0413b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51848/0 (socket says 172.21.15.40:51848) 2023-09-30T00:51:30.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.654+0000 7fec02d9d700 1 -- 172.21.15.40:0/2328403404 learned_addr learned my addr 172.21.15.40:0/2328403404 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:30.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.654+0000 7fec02d9d700 10 monclient(hunting): get_auth_request con 0x7fec0413ac20 auth_method 0 2023-09-30T00:51:30.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.654+0000 7fec02d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:30.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.654+0000 7fec02d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:30.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.654+0000 7fec02d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:30.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.654+0000 7fec02d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:30.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.654+0000 7fec02d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:30.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.654+0000 7fec02d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:30.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.657+0000 7fec02d9d700 10 monclient(hunting): handle_auth_done global_id 4644 payload 274 2023-09-30T00:51:30.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.657+0000 7fec02d9d700 1 -- 172.21.15.40:0/2328403404 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec040b1460 msgr2=0x7fec040b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:30.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.657+0000 7fec02d9d700 1 --2- 172.21.15.40:0/2328403404 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec040b1460 0x7fec040b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:30.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.657+0000 7fec02d9d700 1 -- 172.21.15.40:0/2328403404 >> v1:172.21.15.135:6789/0 conn(0x7fec040a5fb0 legacy=0x7fec04141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:30.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.657+0000 7fec02d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:51:30.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.657+0000 7fec02d9d700 1 monclient: found mon.noname-a 2023-09-30T00:51:30.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.657+0000 7fec02d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:30.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.657+0000 7fec02d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:51:30.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.657+0000 7fec02d9d700 1 -- 172.21.15.40:0/2328403404 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec040a63c0 con 0x7fec0413ac20 2023-09-30T00:51:30.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.676+0000 7fec02d9d700 1 --2- 172.21.15.40:0/2328403404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec0413ac20 0x7fec0413b0a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7febfc009f20 tx=0x7febfc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418c5789c35df414 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:30.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0159a700 1 -- 172.21.15.40:0/2328403404 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc00e070 con 0x7fec0413ac20 2023-09-30T00:51:30.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:30.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:51:30.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0159a700 10 monclient: dump: 2023-09-30T00:51:30.710 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:30.710 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:30.710 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:30.710 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:30.710 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:30.711 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:30.711 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:30.711 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:30.711 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:30.711 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:30.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0159a700 10 monclient: _finish_auth 0 2023-09-30T00:51:30.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:30.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0159a700 1 -- 172.21.15.40:0/2328403404 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febfc00ba80 con 0x7fec0413ac20 2023-09-30T00:51:30.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0896f700 5 monclient: authenticate success, global_id 4644 2023-09-30T00:51:30.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0896f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:51:30.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:30.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0896f700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:30.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0896f700 4 set_mon_vals no callback set 2023-09-30T00:51:30.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0159a700 1 -- 172.21.15.40:0/2328403404 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc0092e0 con 0x7fec0413ac20 2023-09-30T00:51:30.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:30.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:51:30.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.709+0000 7fec0159a700 10 monclient: dump: 2023-09-30T00:51:30.712 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:30.712 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:30.712 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:30.712 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:30.712 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:30.712 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:30.712 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:30.713 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:30.713 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:30.713 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.710+0000 7fec0896f700 10 monclient: shutdown 2023-09-30T00:51:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.710+0000 7fec0896f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.710+0000 7fec0896f700 1 -- 172.21.15.40:0/2328403404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec0413ac20 msgr2=0x7fec0413b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.710+0000 7fec0896f700 1 --2- 172.21.15.40:0/2328403404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec0413ac20 0x7fec0413b0a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7febfc009f20 tx=0x7febfc009840 comp rx=0 tx=0).stop 2023-09-30T00:51:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.710+0000 7fec0896f700 1 -- 172.21.15.40:0/2328403404 shutdown_connections 2023-09-30T00:51:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.710+0000 7fec0896f700 1 --2- 172.21.15.40:0/2328403404 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec0413ac20 0x7fec0413b0a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.710+0000 7fec0896f700 1 --2- 172.21.15.40:0/2328403404 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec040b1460 0x7fec040b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.710+0000 7fec0896f700 1 -- 172.21.15.40:0/2328403404 >> 172.21.15.40:0/2328403404 conn(0x7fec040631f0 msgr2=0x7fec040635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.710+0000 7fec0896f700 1 -- 172.21.15.40:0/2328403404 shutdown_connections 2023-09-30T00:51:30.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.710+0000 7fec0896f700 1 -- 172.21.15.40:0/2328403404 wait complete. 2023-09-30T00:51:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.710+0000 7fec0896f700 10 monclient: build_initial_monmap 2023-09-30T00:51:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.710+0000 7fec0896f700 10 monclient: monmap: 2023-09-30T00:51:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:30.717 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:30.718 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.711+0000 7fec0896f700 1 librados: starting msgr at 2023-09-30T00:51:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.711+0000 7fec0896f700 1 librados: starting objecter 2023-09-30T00:51:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.711+0000 7fec0896f700 1 Processor -- start 2023-09-30T00:51:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.711+0000 7fec0896f700 1 -- start start 2023-09-30T00:51:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.711+0000 7fec0896f700 1 librados: setting wanted keys 2023-09-30T00:51:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.711+0000 7fec0896f700 1 librados: calling monclient init 2023-09-30T00:51:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.711+0000 7fec0896f700 10 monclient: init 2023-09-30T00:51:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.711+0000 7fec0896f700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.711+0000 7fec0896f700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:51:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec0896f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec040a5fb0 0x7fec040aab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec0896f700 10 monclient(hunting): picked mon.noname-a con 0x7fec040a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec0896f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec040b1460 0x7fec0413eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec0896f700 10 monclient(hunting): picked mon.noname-b con 0x7fec040b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec0896f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec0413ac20 0x7fec0414dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec0896f700 10 monclient(hunting): picked mon.noname-c con 0x7fec0413ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec0896f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec0896f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec040ac030 con 0x7fec040a5fb0 2023-09-30T00:51:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec0896f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec0896f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec040aecb0 con 0x7fec040b1460 2023-09-30T00:51:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec0896f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec0896f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec040aa630 con 0x7fec0413ac20 2023-09-30T00:51:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec0896f700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:30.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec0896f700 10 monclient(hunting): authenticate will time out at 5552.013096s 2023-09-30T00:51:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec02d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec0413ac20 0x7fec0414dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.712+0000 7fec01d9b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec040b1460 0x7fec0413eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.713+0000 7fec02d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec0413ac20 0x7fec0414dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:40144/0 (socket says 172.21.15.40:40144) 2023-09-30T00:51:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.713+0000 7fec02d9d700 1 -- 172.21.15.40:0/297775256 learned_addr learned my addr 172.21.15.40:0/297775256 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.713+0000 7fec02d9d700 10 monclient(hunting): get_auth_request con 0x7fec0413ac20 auth_method 0 2023-09-30T00:51:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.713+0000 7fec02d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.713+0000 7fec02d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.713+0000 7fec02d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.713+0000 7fec01d9b700 10 monclient(hunting): get_auth_request con 0x7fec040b1460 auth_method 0 2023-09-30T00:51:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.713+0000 7fec01d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.713+0000 7fec01d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:30.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.713+0000 7fec01d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:30.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.713+0000 7fec01d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:30.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.713+0000 7fec01d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:30.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.713+0000 7fec01d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:30.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.714+0000 7fec02d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:30.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.714+0000 7fec02d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:30.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.714+0000 7fec02d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:30.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec0259c700 1 --2- 172.21.15.40:0/297775256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec040a5fb0 0x7fec040aab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:30.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec0259c700 10 monclient(hunting): get_auth_request con 0x7fec040a5fb0 auth_method 0 2023-09-30T00:51:30.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec0259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:30.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec0259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:30.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec0259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:30.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec01d9b700 10 monclient(hunting): handle_auth_done global_id 4714 payload 824 2023-09-30T00:51:30.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec01d9b700 1 -- 172.21.15.40:0/297775256 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec0413ac20 msgr2=0x7fec0414dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:30.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec01d9b700 1 --2- 172.21.15.40:0/297775256 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec0413ac20 0x7fec0414dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:30.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec01d9b700 1 -- 172.21.15.40:0/297775256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec040a5fb0 msgr2=0x7fec040aab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:30.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec01d9b700 1 --2- 172.21.15.40:0/297775256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec040a5fb0 0x7fec040aab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:30.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec01d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:51:30.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec01d9b700 1 monclient: found mon.noname-b 2023-09-30T00:51:30.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec01d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:30.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec01d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:51:30.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.717+0000 7fec01d9b700 1 -- 172.21.15.40:0/297775256 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc009ae0 con 0x7fec040b1460 2023-09-30T00:51:30.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.719+0000 7fec01d9b700 1 --2- 172.21.15.40:0/297775256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec040b1460 0x7fec0413eb50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7febf800c490 tx=0x7febf8007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:30.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.719+0000 7febf27fc700 10 client.?.objecter ms_handle_connect 0x7fec040b1460 2023-09-30T00:51:30.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.719+0000 7febf27fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:30.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7febf27fc700 1 -- 172.21.15.40:0/297775256 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8011590 con 0x7fec040b1460 2023-09-30T00:51:30.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7febf27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:30.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7febf27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7febf27fc700 10 monclient: dump: 2023-09-30T00:51:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:30.725 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7febf27fc700 10 monclient: _finish_auth 0 2023-09-30T00:51:30.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7febf27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:30.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7febf27fc700 1 -- 172.21.15.40:0/297775256 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf8011710 con 0x7fec040b1460 2023-09-30T00:51:30.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7fec0896f700 5 monclient: authenticate success, global_id 4714 2023-09-30T00:51:30.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7fec0896f700 10 monclient: _renew_subs 2023-09-30T00:51:30.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7fec0896f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:30.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7fec0896f700 1 -- 172.21.15.40:0/297775256 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec040a8d40 con 0x7fec040b1460 2023-09-30T00:51:30.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7febf27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:30.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7febf27fc700 1 -- 172.21.15.40:0/297775256 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8011ad0 con 0x7fec040b1460 2023-09-30T00:51:30.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7febf27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:30.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7febf27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:30.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7febf27fc700 10 monclient: dump: 2023-09-30T00:51:30.727 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:30.727 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:30.727 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:30.727 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:30.727 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:30.727 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:30.727 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:30.727 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:30.727 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:30.727 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:30.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.723+0000 7fec00d99700 4 set_mon_vals no callback set 2023-09-30T00:51:30.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.724+0000 7fec0896f700 10 client.4714.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:30.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.724+0000 7fec0896f700 10 monclient: _renew_subs 2023-09-30T00:51:30.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.724+0000 7fec0896f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:30.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.724+0000 7fec0896f700 1 -- 172.21.15.40:0/297775256 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec040a9fe0 con 0x7fec040b1460 2023-09-30T00:51:30.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.724+0000 7fec0896f700 1 librados: init done 2023-09-30T00:51:30.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.725+0000 7fec0896f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:30.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.725+0000 7fec0896f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:30.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.725+0000 7fec0896f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:30.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.725+0000 7fec0896f700 1 -- 172.21.15.40:0/297775256 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec04064a00 con 0x7fec040b1460 2023-09-30T00:51:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.744+0000 7febf27fc700 1 -- 172.21.15.40:0/297775256 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7febf801b3b0 con 0x7fec040b1460 2023-09-30T00:51:30.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.744+0000 7febf27fc700 1 --2- 172.21.15.40:0/297775256 >> v2:172.21.15.135:6800/116426 conn(0x7febe4068890 0x7febe406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:30.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.745+0000 7fec0259c700 1 --2- 172.21.15.40:0/297775256 >> v2:172.21.15.135:6800/116426 conn(0x7febe4068890 0x7febe406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:30.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.745+0000 7fec0259c700 10 monclient: get_auth_request con 0x7febe4068890 auth_method 0 2023-09-30T00:51:30.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.745+0000 7febf27fc700 10 client.4714.objecter ms_handle_connect 0x7febe4068890 2023-09-30T00:51:30.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.745+0000 7fec0259c700 1 --2- 172.21.15.40:0/297775256 >> v2:172.21.15.135:6800/116426 conn(0x7febe4068890 0x7febe406ad40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7febf4006fd0 tx=0x7febf4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:30.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.976+0000 7febf27fc700 1 -- 172.21.15.40:0/297775256 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7febf8055310 con 0x7fec040b1460 2023-09-30T00:51:30.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.976+0000 7febf27fc700 10 client.4714.objecter ms_dispatch 0x7fec04001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:30.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.976+0000 7febf27fc700 3 client.4714.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:30.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.976+0000 7febf27fc700 3 client.4714.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:30.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.976+0000 7febf27fc700 20 client.4714.objecter dump_active .. 0 homeless 2023-09-30T00:51:30.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.988+0000 7febf27fc700 1 -- 172.21.15.40:0/297775256 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7febf8050d80 con 0x7fec040b1460 2023-09-30T00:51:30.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.988+0000 7febf27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:30.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:30.988+0000 7febf27fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:31.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.241+0000 7fec0896f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:31.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.241+0000 7fec0896f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:31.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.241+0000 7fec0896f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:31.242 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.241+0000 7fec0896f700 1 -- 172.21.15.40:0/297775256 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fec040a56d0 con 0x7fec040b1460 2023-09-30T00:51:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.248+0000 7febf27fc700 1 -- 172.21.15.40:0/297775256 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7febf8017020 con 0x7fec040b1460 2023-09-30T00:51:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.248+0000 7febf27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:31.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.248+0000 7febf27fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:31.248 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:31.248 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:31.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 10 librados: watch_flush enter 2023-09-30T00:51:31.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 10 librados: watch_flush exit 2023-09-30T00:51:31.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 20 client.4714.objecter shutdown clearing up homeless session... 2023-09-30T00:51:31.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 10 client.4714.objecter successfully canceled tick 2023-09-30T00:51:31.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 1 -- 172.21.15.40:0/297775256 >> v2:172.21.15.135:6800/116426 conn(0x7febe4068890 msgr2=0x7febe406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:31.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 1 --2- 172.21.15.40:0/297775256 >> v2:172.21.15.135:6800/116426 conn(0x7febe4068890 0x7febe406ad40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7febf4006fd0 tx=0x7febf4008040 comp rx=0 tx=0).stop 2023-09-30T00:51:31.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 10 monclient: shutdown 2023-09-30T00:51:31.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:31.251 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 1 -- 172.21.15.40:0/297775256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec040b1460 msgr2=0x7fec0413eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:31.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 1 --2- 172.21.15.40:0/297775256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec040b1460 0x7fec0413eb50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7febf800c490 tx=0x7febf8007ec0 comp rx=0 tx=0).stop 2023-09-30T00:51:31.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 1 -- 172.21.15.40:0/297775256 shutdown_connections 2023-09-30T00:51:31.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 1 --2- 172.21.15.40:0/297775256 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fec0413ac20 0x7fec0414dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:31.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 1 --2- 172.21.15.40:0/297775256 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fec040a5fb0 0x7fec040aab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:31.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 1 --2- 172.21.15.40:0/297775256 >> v2:172.21.15.135:6800/116426 conn(0x7febe4068890 0x7febe406ad40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:31.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 1 --2- 172.21.15.40:0/297775256 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fec040b1460 0x7fec0413eb50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:31.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 1 -- 172.21.15.40:0/297775256 >> 172.21.15.40:0/297775256 conn(0x7fec040631f0 msgr2=0x7fec040b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:31.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 1 -- 172.21.15.40:0/297775256 shutdown_connections 2023-09-30T00:51:31.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 1 -- 172.21.15.40:0/297775256 wait complete. 2023-09-30T00:51:31.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:31.250+0000 7fec0896f700 1 librados: shutdown 2023-09-30T00:51:33.741 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:05:09.452 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:33.744 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:05:09.456 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:33.775 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:05:09.486 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:33.797 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:09.484 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:33.800 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:09.487 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:33.801 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:05:09.513 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:33.809 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:05:09.520 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:33.822 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:09.508 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:33.842 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:05:09.554 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:34.265 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:34.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.398+0000 7efed6ebb700 10 monclient: get_monmap_and_config 2023-09-30T00:51:34.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.398+0000 7efed6ebb700 10 monclient: build_initial_monmap 2023-09-30T00:51:34.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.398+0000 7efed6ebb700 10 monclient: monmap: 2023-09-30T00:51:34.399 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:34.399 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:34.399 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:34.399310+0000 2023-09-30T00:51:34.399 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:34.399310+0000 2023-09-30T00:51:34.399 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:34.399 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:34.399 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:34.399 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:34.399 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:34.399 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:34.400 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:34.400 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 1 Processor -- start 2023-09-30T00:51:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 1 -- start start 2023-09-30T00:51:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 10 monclient: init 2023-09-30T00:51:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-09-30T00:51:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00b1460 0x7efed00b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 10 monclient(hunting): picked mon.noname-b con 0x7efed00b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efed00a5fb0 addr v1:172.21.15.40:6789/0 2023-09-30T00:51:34.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efed013ac20 0x7efed013b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 10 monclient(hunting): picked mon.noname-a con 0x7efed013ac20 addr v2:172.21.15.40:3300/0 2023-09-30T00:51:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efed00a5b20 con 0x7efed00a5fb0 2023-09-30T00:51:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efed00a71f0 con 0x7efed013ac20 2023-09-30T00:51:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed00a74d0 con 0x7efed00b1460 2023-09-30T00:51:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.399+0000 7efed6ebb700 10 monclient(hunting): authenticate will time out at 5555.700932s 2023-09-30T00:51:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.400+0000 7efed4c57700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00b1460 0x7efed00b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.400+0000 7efed5458700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efed013ac20 0x7efed013b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed4c57700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00b1460 0x7efed00b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:52800/0 (socket says 172.21.15.40:52800) 2023-09-30T00:51:34.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed4c57700 1 -- 172.21.15.40:0/2441026180 learned_addr learned my addr 172.21.15.40:0/2441026180 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed4c57700 10 monclient(hunting): get_auth_request con 0x7efed00b1460 auth_method 0 2023-09-30T00:51:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed4c57700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed4c57700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed4c57700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed5458700 10 monclient(hunting): get_auth_request con 0x7efed013ac20 auth_method 0 2023-09-30T00:51:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed5458700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed5458700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed5458700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed4c57700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed4c57700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed4c57700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:34.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed5458700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:34.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed5458700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:34.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.401+0000 7efed5458700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:34.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.404+0000 7efed4c57700 10 monclient(hunting): handle_auth_done global_id 4577 payload 274 2023-09-30T00:51:34.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.404+0000 7efed4c57700 1 -- 172.21.15.40:0/2441026180 >> v2:172.21.15.40:3300/0 conn(0x7efed013ac20 msgr2=0x7efed013b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:34.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.404+0000 7efed4c57700 1 --2- 172.21.15.40:0/2441026180 >> v2:172.21.15.40:3300/0 conn(0x7efed013ac20 0x7efed013b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:34.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.404+0000 7efed4c57700 1 -- 172.21.15.40:0/2441026180 >> v1:172.21.15.40:6789/0 conn(0x7efed00a5fb0 legacy=0x7efed0141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:34.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.404+0000 7efed4c57700 10 monclient: _finish_hunting 0 2023-09-30T00:51:34.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.404+0000 7efed4c57700 1 monclient: found mon.noname-b 2023-09-30T00:51:34.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.404+0000 7efed4c57700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:34.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.405+0000 7efed4c57700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:51:34.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.405+0000 7efed4c57700 1 -- 172.21.15.40:0/2441026180 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed00a63c0 con 0x7efed00b1460 2023-09-30T00:51:34.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.405+0000 7efed5458700 1 --2- 172.21.15.40:0/2441026180 >> v2:172.21.15.40:3300/0 conn(0x7efed013ac20 0x7efed013b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:51:34.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.407+0000 7efed4c57700 1 --2- 172.21.15.40:0/2441026180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00b1460 0x7efed00b1830 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efec00099b0 tx=0x7efec0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=642e6825b4b6fa58 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:34.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efecf7fe700 1 -- 172.21.15.40:0/2441026180 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec0009ea0 con 0x7efed00b1460 2023-09-30T00:51:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efecf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efecf7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efecf7fe700 10 monclient: dump: 2023-09-30T00:51:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:34.413 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:34.414 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efecf7fe700 10 monclient: _finish_auth 0 2023-09-30T00:51:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efecf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efecf7fe700 1 -- 172.21.15.40:0/2441026180 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec000b820 con 0x7efed00b1460 2023-09-30T00:51:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efecf7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efecf7fe700 1 -- 172.21.15.40:0/2441026180 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec000bbc0 con 0x7efed00b1460 2023-09-30T00:51:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efed6ebb700 5 monclient: authenticate success, global_id 4577 2023-09-30T00:51:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efed6ebb700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efed6ebb700 4 set_mon_vals no callback set 2023-09-30T00:51:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efecf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:34.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efecf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:51:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.412+0000 7efecf7fe700 10 monclient: dump: 2023-09-30T00:51:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:34.415 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:34.415 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:34.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.413+0000 7efed6ebb700 10 monclient: shutdown 2023-09-30T00:51:34.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.413+0000 7efed6ebb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:34.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.413+0000 7efed6ebb700 1 -- 172.21.15.40:0/2441026180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00b1460 msgr2=0x7efed00b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:34.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.413+0000 7efed6ebb700 1 --2- 172.21.15.40:0/2441026180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00b1460 0x7efed00b1830 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efec00099b0 tx=0x7efec0009250 comp rx=0 tx=0).stop 2023-09-30T00:51:34.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.413+0000 7efed6ebb700 1 -- 172.21.15.40:0/2441026180 shutdown_connections 2023-09-30T00:51:34.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.413+0000 7efed6ebb700 1 --2- 172.21.15.40:0/2441026180 >> v2:172.21.15.40:3300/0 conn(0x7efed013ac20 0x7efed013b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:34.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.413+0000 7efed6ebb700 1 --2- 172.21.15.40:0/2441026180 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00b1460 0x7efed00b1830 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:34.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.413+0000 7efed6ebb700 1 -- 172.21.15.40:0/2441026180 >> 172.21.15.40:0/2441026180 conn(0x7efed00631f0 msgr2=0x7efed00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:34.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.413+0000 7efed6ebb700 1 -- 172.21.15.40:0/2441026180 shutdown_connections 2023-09-30T00:51:34.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.413+0000 7efed6ebb700 1 -- 172.21.15.40:0/2441026180 wait complete. 2023-09-30T00:51:34.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.413+0000 7efed6ebb700 10 monclient: build_initial_monmap 2023-09-30T00:51:34.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.413+0000 7efed6ebb700 10 monclient: monmap: 2023-09-30T00:51:34.420 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:34.420 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:34.420 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:34.420 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:34.420 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:34.420 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:34.420 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:34.420 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:34.420 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:34.420 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:34.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.414+0000 7efed6ebb700 1 librados: starting msgr at 2023-09-30T00:51:34.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.414+0000 7efed6ebb700 1 librados: starting objecter 2023-09-30T00:51:34.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.414+0000 7efed6ebb700 1 Processor -- start 2023-09-30T00:51:34.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.414+0000 7efed6ebb700 1 -- start start 2023-09-30T00:51:34.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.414+0000 7efed6ebb700 1 librados: setting wanted keys 2023-09-30T00:51:34.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.414+0000 7efed6ebb700 1 librados: calling monclient init 2023-09-30T00:51:34.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.414+0000 7efed6ebb700 10 monclient: init 2023-09-30T00:51:34.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:34.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:51:34.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efed00a5fb0 0x7efed00a9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:34.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 10 monclient(hunting): picked mon.noname-b con 0x7efed00a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:34.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00b1460 0x7efed00a8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:34.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 10 monclient(hunting): picked mon.noname-c con 0x7efed00b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed013ac20 0x7efed01388b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 10 monclient(hunting): picked mon.noname-a con 0x7efed013ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed00a9690 con 0x7efed013ac20 2023-09-30T00:51:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efed00aa6d0 con 0x7efed00a5fb0 2023-09-30T00:51:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efed00a88b0 con 0x7efed00b1460 2023-09-30T00:51:34.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed6ebb700 10 monclient(hunting): authenticate will time out at 5555.716224s 2023-09-30T00:51:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efed5458700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed013ac20 0x7efed01388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.415+0000 7efecffff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00b1460 0x7efed00a8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efed4c57700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efed00a5fb0 0x7efed00a9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efed5458700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed013ac20 0x7efed01388b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47288/0 (socket says 172.21.15.40:47288) 2023-09-30T00:51:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efed5458700 1 -- 172.21.15.40:0/3681217619 learned_addr learned my addr 172.21.15.40:0/3681217619 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efed5458700 10 monclient(hunting): get_auth_request con 0x7efed013ac20 auth_method 0 2023-09-30T00:51:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efed5458700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efed5458700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efed5458700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efecffff700 10 monclient(hunting): get_auth_request con 0x7efed00b1460 auth_method 0 2023-09-30T00:51:34.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efecffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efecffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efecffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efed4c57700 10 monclient(hunting): get_auth_request con 0x7efed00a5fb0 auth_method 0 2023-09-30T00:51:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efed4c57700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efed4c57700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.416+0000 7efed4c57700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.417+0000 7efed5458700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.417+0000 7efed5458700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.417+0000 7efed5458700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.417+0000 7efecffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.417+0000 7efecffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.417+0000 7efecffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:34.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.417+0000 7efed4c57700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.417+0000 7efed4c57700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:34.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.417+0000 7efed4c57700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.420+0000 7efecffff700 10 monclient(hunting): handle_auth_done global_id 4580 payload 824 2023-09-30T00:51:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.421+0000 7efecffff700 1 -- 172.21.15.40:0/3681217619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efed00a5fb0 msgr2=0x7efed00a9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.421+0000 7efecffff700 1 --2- 172.21.15.40:0/3681217619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efed00a5fb0 0x7efed00a9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.421+0000 7efecffff700 1 -- 172.21.15.40:0/3681217619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed013ac20 msgr2=0x7efed01388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.421+0000 7efecffff700 1 --2- 172.21.15.40:0/3681217619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed013ac20 0x7efed01388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.421+0000 7efecffff700 10 monclient: _finish_hunting 0 2023-09-30T00:51:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.421+0000 7efecffff700 1 monclient: found mon.noname-c 2023-09-30T00:51:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.421+0000 7efecffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.421+0000 7efecffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:51:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.421+0000 7efecffff700 1 -- 172.21.15.40:0/3681217619 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec8009a20 con 0x7efed00b1460 2023-09-30T00:51:34.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.423+0000 7efecffff700 1 --2- 172.21.15.40:0/3681217619 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00b1460 0x7efed00a8000 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efeb800b510 tx=0x7efeb8009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.423+0000 7efeccff9700 10 client.?.objecter ms_handle_connect 0x7efed00b1460 2023-09-30T00:51:34.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.423+0000 7efeccff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:34.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efeccff9700 1 -- 172.21.15.40:0/3681217619 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb800be60 con 0x7efed00b1460 2023-09-30T00:51:34.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efeccff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:34.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efeccff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:51:34.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efeccff9700 10 monclient: dump: 2023-09-30T00:51:34.428 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:34.428 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:34.429 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efeccff9700 10 monclient: _finish_auth 0 2023-09-30T00:51:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efeccff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efeccff9700 1 -- 172.21.15.40:0/3681217619 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb800f510 con 0x7efed00b1460 2023-09-30T00:51:34.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efeccff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efeccff9700 1 -- 172.21.15.40:0/3681217619 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb800f8b0 con 0x7efed00b1460 2023-09-30T00:51:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efed6ebb700 5 monclient: authenticate success, global_id 4580 2023-09-30T00:51:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efeccff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efeccff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:51:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efeccff9700 10 monclient: dump: 2023-09-30T00:51:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:34.430 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:34.431 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efed6ebb700 10 monclient: _renew_subs 2023-09-30T00:51:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efeceffd700 4 set_mon_vals no callback set 2023-09-30T00:51:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efed6ebb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efed6ebb700 1 -- 172.21.15.40:0/3681217619 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec00094f0 con 0x7efed00b1460 2023-09-30T00:51:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efed6ebb700 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efed6ebb700 10 monclient: _renew_subs 2023-09-30T00:51:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efed6ebb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efed6ebb700 1 -- 172.21.15.40:0/3681217619 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efed00a8540 con 0x7efed00b1460 2023-09-30T00:51:34.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.427+0000 7efed6ebb700 1 librados: init done 2023-09-30T00:51:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.428+0000 7efed6ebb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.428+0000 7efed6ebb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.428+0000 7efed6ebb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:34.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.428+0000 7efed6ebb700 1 -- 172.21.15.40:0/3681217619 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efed013d150 con 0x7efed00b1460 2023-09-30T00:51:34.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.747+0000 7efeccff9700 1 -- 172.21.15.40:0/3681217619 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7efeb800fc20 con 0x7efed00b1460 2023-09-30T00:51:34.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.748+0000 7efeccff9700 1 --2- 172.21.15.40:0/3681217619 >> v2:172.21.15.135:6800/116426 conn(0x7efeb0068960 0x7efeb006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:34.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.748+0000 7efeccff9700 1 -- 172.21.15.40:0/3681217619 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7efeb8084100 con 0x7efed00b1460 2023-09-30T00:51:34.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.748+0000 7efeccff9700 10 client.4580.objecter ms_dispatch 0x7efed0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:34.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.748+0000 7efeccff9700 3 client.4580.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:34.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.748+0000 7efeccff9700 3 client.4580.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:34.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.748+0000 7efeccff9700 20 client.4580.objecter dump_active .. 0 homeless 2023-09-30T00:51:34.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.748+0000 7efeccff9700 1 -- 172.21.15.40:0/3681217619 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efeb80844d0 con 0x7efed00b1460 2023-09-30T00:51:34.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.748+0000 7efeccff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:34.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.748+0000 7efeccff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:34.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.748+0000 7efed4c57700 1 --2- 172.21.15.40:0/3681217619 >> v2:172.21.15.135:6800/116426 conn(0x7efeb0068960 0x7efeb006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:34.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.748+0000 7efed4c57700 10 monclient: get_auth_request con 0x7efeb0068960 auth_method 0 2023-09-30T00:51:34.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.749+0000 7efeccff9700 10 client.4580.objecter ms_handle_connect 0x7efeb0068960 2023-09-30T00:51:34.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:34.749+0000 7efed4c57700 1 --2- 172.21.15.40:0/3681217619 >> v2:172.21.15.135:6800/116426 conn(0x7efeb0068960 0x7efeb006ae10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efec00099b0 tx=0x7efec0016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:35.008 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.007+0000 7efed6ebb700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:35.008 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.007+0000 7efed6ebb700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:35.008 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.007+0000 7efed6ebb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:35.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.007+0000 7efed6ebb700 1 -- 172.21.15.40:0/3681217619 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efed013d150 con 0x7efed00b1460 2023-09-30T00:51:35.066 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.066+0000 7efeccff9700 1 -- 172.21.15.40:0/3681217619 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7efed013d150 con 0x7efed00b1460 2023-09-30T00:51:35.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.066+0000 7efeccff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:35.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.066+0000 7efeccff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:35.067 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:35.067 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:35.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 10 librados: watch_flush enter 2023-09-30T00:51:35.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 10 librados: watch_flush exit 2023-09-30T00:51:35.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 20 client.4580.objecter shutdown clearing up homeless session... 2023-09-30T00:51:35.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 10 client.4580.objecter successfully canceled tick 2023-09-30T00:51:35.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 1 -- 172.21.15.40:0/3681217619 >> v2:172.21.15.135:6800/116426 conn(0x7efeb0068960 msgr2=0x7efeb006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:35.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 1 --2- 172.21.15.40:0/3681217619 >> v2:172.21.15.135:6800/116426 conn(0x7efeb0068960 0x7efeb006ae10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efec00099b0 tx=0x7efec0016000 comp rx=0 tx=0).stop 2023-09-30T00:51:35.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 10 monclient: shutdown 2023-09-30T00:51:35.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:35.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 1 -- 172.21.15.40:0/3681217619 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00b1460 msgr2=0x7efed00a8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:35.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 1 --2- 172.21.15.40:0/3681217619 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00b1460 0x7efed00a8000 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efeb800b510 tx=0x7efeb8009060 comp rx=0 tx=0).stop 2023-09-30T00:51:35.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 1 -- 172.21.15.40:0/3681217619 shutdown_connections 2023-09-30T00:51:35.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 1 --2- 172.21.15.40:0/3681217619 >> v2:172.21.15.135:6800/116426 conn(0x7efeb0068960 0x7efeb006ae10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:35.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 1 --2- 172.21.15.40:0/3681217619 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efed00a5fb0 0x7efed00a9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:35.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 1 --2- 172.21.15.40:0/3681217619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efed013ac20 0x7efed01388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:35.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 1 --2- 172.21.15.40:0/3681217619 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efed00b1460 0x7efed00a8000 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:35.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.068+0000 7efed6ebb700 1 -- 172.21.15.40:0/3681217619 >> 172.21.15.40:0/3681217619 conn(0x7efed00631f0 msgr2=0x7efed00b5180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:35.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.069+0000 7efed6ebb700 1 -- 172.21.15.40:0/3681217619 shutdown_connections 2023-09-30T00:51:35.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.069+0000 7efed6ebb700 1 -- 172.21.15.40:0/3681217619 wait complete. 2023-09-30T00:51:35.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:35.069+0000 7efed6ebb700 1 librados: shutdown 2023-09-30T00:51:38.084 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:38.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.230+0000 7f9b936e1700 10 monclient: get_monmap_and_config 2023-09-30T00:51:38.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.230+0000 7f9b936e1700 10 monclient: build_initial_monmap 2023-09-30T00:51:38.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.230+0000 7f9b936e1700 10 monclient: monmap: 2023-09-30T00:51:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:38.231087+0000 2023-09-30T00:51:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:38.231087+0000 2023-09-30T00:51:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:38.231 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:38.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.230+0000 7f9b936e1700 1 Processor -- start 2023-09-30T00:51:38.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.230+0000 7f9b936e1700 1 -- start start 2023-09-30T00:51:38.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.230+0000 7f9b936e1700 10 monclient: init 2023-09-30T00:51:38.233 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.231+0000 7f9b936e1700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.231+0000 7f9b936e1700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-09-30T00:51:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.231+0000 7f9b936e1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b8c0b16a0 addr v1:172.21.15.40:6789/0 2023-09-30T00:51:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.231+0000 7f9b936e1700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9b8c0a54c0 0x7f9b8c0a5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.231+0000 7f9b936e1700 10 monclient(hunting): picked mon.noname-a con 0x7f9b8c0a54c0 addr v2:172.21.15.40:3300/0 2023-09-30T00:51:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.231+0000 7f9b936e1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b8c13aeb0 0x7f9b8c13b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.231+0000 7f9b936e1700 10 monclient(hunting): picked mon.noname-b con 0x7f9b8c13aeb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.231+0000 7f9b936e1700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b8c0a7f00 con 0x7f9b8c0b16a0 2023-09-30T00:51:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.231+0000 7f9b936e1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.231+0000 7f9b936e1700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b8c0a8390 con 0x7f9b8c0a54c0 2023-09-30T00:51:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.231+0000 7f9b936e1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.231+0000 7f9b936e1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c0b1a70 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.231+0000 7f9b936e1700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.231+0000 7f9b936e1700 10 monclient(hunting): authenticate will time out at 5559.532609s 2023-09-30T00:51:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.232+0000 7f9b91c7e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b8c13aeb0 0x7f9b8c13b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.232+0000 7f9b90c7c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9b8c0a54c0 0x7f9b8c0a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.232+0000 7f9b91c7e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b8c13aeb0 0x7f9b8c13b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:52810/0 (socket says 172.21.15.40:52810) 2023-09-30T00:51:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.232+0000 7f9b91c7e700 1 -- 172.21.15.40:0/1995482444 learned_addr learned my addr 172.21.15.40:0/1995482444 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.232+0000 7f9b91c7e700 10 monclient(hunting): get_auth_request con 0x7f9b8c13aeb0 auth_method 0 2023-09-30T00:51:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.232+0000 7f9b91c7e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.232+0000 7f9b91c7e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.232+0000 7f9b91c7e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.233+0000 7f9b90c7c700 10 monclient(hunting): get_auth_request con 0x7f9b8c0a54c0 auth_method 0 2023-09-30T00:51:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.233+0000 7f9b90c7c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.233+0000 7f9b90c7c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:38.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.233+0000 7f9b90c7c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:38.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.233+0000 7f9b91c7e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:38.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.233+0000 7f9b91c7e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:38.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.233+0000 7f9b91c7e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:38.236 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.234+0000 7f9b90c7c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.234+0000 7f9b90c7c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.234+0000 7f9b90c7c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.236+0000 7f9b91c7e700 10 monclient(hunting): handle_auth_done global_id 4583 payload 274 2023-09-30T00:51:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.236+0000 7f9b91c7e700 1 -- 172.21.15.40:0/1995482444 >> v2:172.21.15.40:3300/0 conn(0x7f9b8c0a54c0 msgr2=0x7f9b8c0a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.236+0000 7f9b91c7e700 1 --2- 172.21.15.40:0/1995482444 >> v2:172.21.15.40:3300/0 conn(0x7f9b8c0a54c0 0x7f9b8c0a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.236+0000 7f9b91c7e700 1 -- 172.21.15.40:0/1995482444 >> v1:172.21.15.40:6789/0 conn(0x7f9b8c0b16a0 legacy=0x7f9b8c0a7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.236+0000 7f9b91c7e700 10 monclient: _finish_hunting 0 2023-09-30T00:51:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.236+0000 7f9b91c7e700 1 monclient: found mon.noname-b 2023-09-30T00:51:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.236+0000 7f9b91c7e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.236+0000 7f9b91c7e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:51:38.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.236+0000 7f9b91c7e700 1 -- 172.21.15.40:0/1995482444 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c0a6f30 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.241+0000 7f9b91c7e700 1 --2- 172.21.15.40:0/1995482444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b8c13aeb0 0x7f9b8c13b310 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009f20 tx=0x7f9b84009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bda50b1e46719bc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:38.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b83fff700 1 -- 172.21.15.40:0/1995482444 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8400e070 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b83fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:38.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b83fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:38.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b83fff700 10 monclient: dump: 2023-09-30T00:51:38.247 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:38.247 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:38.247 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:38.247 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:38.247 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:38.247 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:38.247 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:38.247 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:38.247 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:38.247 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:38.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b83fff700 10 monclient: _finish_auth 0 2023-09-30T00:51:38.247 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b83fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b83fff700 1 -- 172.21.15.40:0/1995482444 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8400b9d0 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b83fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b83fff700 1 -- 172.21.15.40:0/1995482444 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b840092e0 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b936e1700 5 monclient: authenticate success, global_id 4583 2023-09-30T00:51:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b936e1700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b83fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b936e1700 4 set_mon_vals no callback set 2023-09-30T00:51:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b83fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:51:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b83fff700 10 monclient: dump: 2023-09-30T00:51:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:38.248 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:38.249 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:38.249 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:38.249 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:38.249 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:38.249 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:38.249 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:38.249 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b936e1700 10 monclient: shutdown 2023-09-30T00:51:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b936e1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b936e1700 1 -- 172.21.15.40:0/1995482444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b8c13aeb0 msgr2=0x7f9b8c13b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b936e1700 1 --2- 172.21.15.40:0/1995482444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b8c13aeb0 0x7f9b8c13b310 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009f20 tx=0x7f9b84009840 comp rx=0 tx=0).stop 2023-09-30T00:51:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b936e1700 1 -- 172.21.15.40:0/1995482444 shutdown_connections 2023-09-30T00:51:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b936e1700 1 --2- 172.21.15.40:0/1995482444 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b8c13aeb0 0x7f9b8c13b310 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b936e1700 1 --2- 172.21.15.40:0/1995482444 >> v2:172.21.15.40:3300/0 conn(0x7f9b8c0a54c0 0x7f9b8c0a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.246+0000 7f9b936e1700 1 -- 172.21.15.40:0/1995482444 >> 172.21.15.40:0/1995482444 conn(0x7f9b8c06f130 msgr2=0x7f9b8c0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:38.252 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.247+0000 7f9b936e1700 1 -- 172.21.15.40:0/1995482444 shutdown_connections 2023-09-30T00:51:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.247+0000 7f9b936e1700 1 -- 172.21.15.40:0/1995482444 wait complete. 2023-09-30T00:51:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.247+0000 7f9b936e1700 10 monclient: build_initial_monmap 2023-09-30T00:51:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.248+0000 7f9b936e1700 10 monclient: monmap: 2023-09-30T00:51:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:38.254 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.248+0000 7f9b936e1700 1 librados: starting msgr at 2023-09-30T00:51:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.248+0000 7f9b936e1700 1 librados: starting objecter 2023-09-30T00:51:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.248+0000 7f9b936e1700 1 Processor -- start 2023-09-30T00:51:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.248+0000 7f9b936e1700 1 -- start start 2023-09-30T00:51:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.248+0000 7f9b936e1700 1 librados: setting wanted keys 2023-09-30T00:51:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.248+0000 7f9b936e1700 1 librados: calling monclient init 2023-09-30T00:51:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.248+0000 7f9b936e1700 10 monclient: init 2023-09-30T00:51:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:51:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b8c0a54c0 0x7f9b8c13ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:38.254 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 10 monclient(hunting): picked mon.noname-a con 0x7f9b8c0a54c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b8c0b16a0 0x7f9b8c0aa740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 10 monclient(hunting): picked mon.noname-c con 0x7f9b8c0b16a0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b8c13aeb0 0x7f9b8c13e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 10 monclient(hunting): picked mon.noname-b con 0x7f9b8c13aeb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c1404e0 con 0x7f9b8c0a54c0 2023-09-30T00:51:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c0aac80 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8c145080 con 0x7f9b8c0b16a0 2023-09-30T00:51:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:38.255 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.249+0000 7f9b936e1700 10 monclient(hunting): authenticate will time out at 5559.550784s 2023-09-30T00:51:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.250+0000 7f9b91c7e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b8c13aeb0 0x7f9b8c13e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.250+0000 7f9b9147d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b8c0a54c0 0x7f9b8c13ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.250+0000 7f9b9147d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b8c0a54c0 0x7f9b8c13ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47308/0 (socket says 172.21.15.40:47308) 2023-09-30T00:51:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.250+0000 7f9b9147d700 1 -- 172.21.15.40:0/3268344926 learned_addr learned my addr 172.21.15.40:0/3268344926 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.250+0000 7f9b9147d700 10 monclient(hunting): get_auth_request con 0x7f9b8c0a54c0 auth_method 0 2023-09-30T00:51:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.250+0000 7f9b9147d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.250+0000 7f9b9147d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.250+0000 7f9b9147d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.250+0000 7f9b91c7e700 10 monclient(hunting): get_auth_request con 0x7f9b8c13aeb0 auth_method 0 2023-09-30T00:51:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.250+0000 7f9b91c7e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.250+0000 7f9b91c7e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.250+0000 7f9b91c7e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.251+0000 7f9b9147d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:38.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.251+0000 7f9b9147d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:38.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.251+0000 7f9b9147d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:38.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.251+0000 7f9b91c7e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:38.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.251+0000 7f9b91c7e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:38.257 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.251+0000 7f9b91c7e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:38.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.253+0000 7f9b90c7c700 1 --2- 172.21.15.40:0/3268344926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b8c0b16a0 0x7f9b8c0aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:38.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.254+0000 7f9b90c7c700 10 monclient(hunting): get_auth_request con 0x7f9b8c0b16a0 auth_method 0 2023-09-30T00:51:38.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.254+0000 7f9b90c7c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:38.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.254+0000 7f9b90c7c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:38.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.254+0000 7f9b90c7c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:38.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.254+0000 7f9b90c7c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:38.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.254+0000 7f9b90c7c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:38.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.254+0000 7f9b90c7c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:38.258 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.255+0000 7f9b91c7e700 10 monclient(hunting): handle_auth_done global_id 4720 payload 824 2023-09-30T00:51:38.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.255+0000 7f9b91c7e700 1 -- 172.21.15.40:0/3268344926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b8c0b16a0 msgr2=0x7f9b8c0aa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:38.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.255+0000 7f9b91c7e700 1 --2- 172.21.15.40:0/3268344926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b8c0b16a0 0x7f9b8c0aa740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:38.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.255+0000 7f9b91c7e700 1 -- 172.21.15.40:0/3268344926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b8c0a54c0 msgr2=0x7f9b8c13ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:38.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.255+0000 7f9b91c7e700 1 --2- 172.21.15.40:0/3268344926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b8c0a54c0 0x7f9b8c13ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:38.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.255+0000 7f9b91c7e700 10 monclient: _finish_hunting 0 2023-09-30T00:51:38.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.255+0000 7f9b91c7e700 1 monclient: found mon.noname-b 2023-09-30T00:51:38.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.255+0000 7f9b91c7e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:38.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.255+0000 7f9b91c7e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:51:38.259 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.255+0000 7f9b91c7e700 1 -- 172.21.15.40:0/3268344926 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b84009ae0 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.260 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.258+0000 7f9b91c7e700 1 --2- 172.21.15.40:0/3268344926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b8c13aeb0 0x7f9b8c13e7f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9b84015000 tx=0x7f9b84018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:38.260 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.258+0000 7f9b817fa700 10 client.?.objecter ms_handle_connect 0x7f9b8c13aeb0 2023-09-30T00:51:38.260 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.258+0000 7f9b817fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:38.263 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b817fa700 1 -- 172.21.15.40:0/3268344926 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84007ac0 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b817fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:38.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b817fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:38.264 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b817fa700 10 monclient: dump: 2023-09-30T00:51:38.264 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:38.264 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:38.264 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:38.264 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:38.264 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:38.264 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:38.264 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:38.264 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:38.264 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:38.265 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:38.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b817fa700 10 monclient: _finish_auth 0 2023-09-30T00:51:38.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b817fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:38.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b817fa700 1 -- 172.21.15.40:0/3268344926 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b84007c20 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b817fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:38.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b817fa700 1 -- 172.21.15.40:0/3268344926 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8401d420 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b936e1700 5 monclient: authenticate success, global_id 4720 2023-09-30T00:51:38.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b936e1700 10 monclient: _renew_subs 2023-09-30T00:51:38.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b936e1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:38.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b936e1700 1 -- 172.21.15.40:0/3268344926 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8c13bf50 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b82ffd700 4 set_mon_vals no callback set 2023-09-30T00:51:38.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b817fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:38.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b817fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:38.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b817fa700 10 monclient: dump: 2023-09-30T00:51:38.266 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:38.266 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:38.266 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:38.266 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:38.266 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:38.266 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:38.266 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:38.266 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:38.266 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:38.266 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:38.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b936e1700 10 client.4720.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:38.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b936e1700 10 monclient: _renew_subs 2023-09-30T00:51:38.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b936e1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:38.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b936e1700 1 -- 172.21.15.40:0/3268344926 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b8c13ef70 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.263+0000 7f9b936e1700 1 librados: init done 2023-09-30T00:51:38.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.264+0000 7f9b936e1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:38.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.264+0000 7f9b936e1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:38.267 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.264+0000 7f9b936e1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:38.268 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.264+0000 7f9b936e1700 1 -- 172.21.15.40:0/3268344926 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b8c13f100 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.284 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.284+0000 7f9b817fa700 1 -- 172.21.15.40:0/3268344926 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9b84003040 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.285 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.284+0000 7f9b817fa700 1 --2- 172.21.15.40:0/3268344926 >> v2:172.21.15.135:6800/116426 conn(0x7f9b70068890 0x7f9b7006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:38.285 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.284+0000 7f9b9147d700 1 --2- 172.21.15.40:0/3268344926 >> v2:172.21.15.135:6800/116426 conn(0x7f9b70068890 0x7f9b7006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:38.285 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.284+0000 7f9b9147d700 10 monclient: get_auth_request con 0x7f9b70068890 auth_method 0 2023-09-30T00:51:38.285 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.285+0000 7f9b817fa700 10 client.4720.objecter ms_handle_connect 0x7f9b70068890 2023-09-30T00:51:38.285 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.285+0000 7f9b9147d700 1 --2- 172.21.15.40:0/3268344926 >> v2:172.21.15.135:6800/116426 conn(0x7f9b70068890 0x7f9b7006ad40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9b880078a0 tx=0x7f9b88007830 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.519+0000 7f9b817fa700 1 -- 172.21.15.40:0/3268344926 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9b84057960 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.519+0000 7f9b817fa700 10 client.4720.objecter ms_dispatch 0x7f9b8c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.519+0000 7f9b817fa700 3 client.4720.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:38.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.519+0000 7f9b817fa700 3 client.4720.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:38.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.519+0000 7f9b817fa700 20 client.4720.objecter dump_active .. 0 homeless 2023-09-30T00:51:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.529+0000 7f9b817fa700 1 -- 172.21.15.40:0/3268344926 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9b840535a0 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.529 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.529+0000 7f9b817fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:38.530 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.529+0000 7f9b817fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:38.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.798+0000 7f9b936e1700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:38.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.798+0000 7f9b936e1700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:38.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.798+0000 7f9b936e1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:38.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.798+0000 7f9b936e1700 1 -- 172.21.15.40:0/3268344926 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9b8c072b90 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.836+0000 7f9b817fa700 1 -- 172.21.15.40:0/3268344926 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9b84056bc0 con 0x7f9b8c13aeb0 2023-09-30T00:51:38.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.836+0000 7f9b817fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:38.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.836+0000 7f9b817fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:38.837 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:38.837 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 10 librados: watch_flush enter 2023-09-30T00:51:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 10 librados: watch_flush exit 2023-09-30T00:51:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 20 client.4720.objecter shutdown clearing up homeless session... 2023-09-30T00:51:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 10 client.4720.objecter successfully canceled tick 2023-09-30T00:51:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 1 -- 172.21.15.40:0/3268344926 >> v2:172.21.15.135:6800/116426 conn(0x7f9b70068890 msgr2=0x7f9b7006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 1 --2- 172.21.15.40:0/3268344926 >> v2:172.21.15.135:6800/116426 conn(0x7f9b70068890 0x7f9b7006ad40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9b880078a0 tx=0x7f9b88007830 comp rx=0 tx=0).stop 2023-09-30T00:51:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 10 monclient: shutdown 2023-09-30T00:51:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:38.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 1 -- 172.21.15.40:0/3268344926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b8c13aeb0 msgr2=0x7f9b8c13e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:38.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 1 --2- 172.21.15.40:0/3268344926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b8c13aeb0 0x7f9b8c13e7f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9b84015000 tx=0x7f9b84018000 comp rx=0 tx=0).stop 2023-09-30T00:51:38.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 1 -- 172.21.15.40:0/3268344926 shutdown_connections 2023-09-30T00:51:38.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 1 --2- 172.21.15.40:0/3268344926 >> v2:172.21.15.135:6800/116426 conn(0x7f9b70068890 0x7f9b7006ad40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:38.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 1 --2- 172.21.15.40:0/3268344926 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b8c13aeb0 0x7f9b8c13e7f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:38.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 1 --2- 172.21.15.40:0/3268344926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b8c0a54c0 0x7f9b8c13ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:38.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 1 --2- 172.21.15.40:0/3268344926 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b8c0b16a0 0x7f9b8c0aa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:38.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 1 -- 172.21.15.40:0/3268344926 >> 172.21.15.40:0/3268344926 conn(0x7f9b8c06f130 msgr2=0x7f9b8c0b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:38.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 1 -- 172.21.15.40:0/3268344926 shutdown_connections 2023-09-30T00:51:38.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 1 -- 172.21.15.40:0/3268344926 wait complete. 2023-09-30T00:51:38.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:38.839+0000 7f9b936e1700 1 librados: shutdown 2023-09-30T00:51:41.854 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:42.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.006+0000 7f3734db7700 10 monclient: get_monmap_and_config 2023-09-30T00:51:42.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.006+0000 7f3734db7700 10 monclient: build_initial_monmap 2023-09-30T00:51:42.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.006+0000 7f3734db7700 10 monclient: monmap: 2023-09-30T00:51:42.008 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:42.008 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:42.008 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:42.007417+0000 2023-09-30T00:51:42.008 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:42.007417+0000 2023-09-30T00:51:42.008 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:42.008 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:42.009 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:42.009 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:42.009 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:42.009 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:42.009 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:42.009 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:42.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.007+0000 7f3734db7700 1 Processor -- start 2023-09-30T00:51:42.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.007+0000 7f3734db7700 1 -- start start 2023-09-30T00:51:42.009 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.007+0000 7f3734db7700 10 monclient: init 2023-09-30T00:51:42.011 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.007+0000 7f3734db7700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:42.011 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.007+0000 7f3734db7700 10 monclient: _add_conns ranks=[0,4,1,3,2] 2023-09-30T00:51:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.008+0000 7f3734db7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37300b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:51:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.008+0000 7f3734db7700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f37300b1830 0x7f37300a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.008+0000 7f3734db7700 10 monclient(hunting): picked mon.noname-c con 0x7f37300b1830 addr v2:172.21.15.135:3300/0 2023-09-30T00:51:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.008+0000 7f3734db7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f373013ac20 addr v1:172.21.15.135:6789/0 2023-09-30T00:51:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.008+0000 7f3734db7700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37300a5b20 con 0x7f37300b1460 2023-09-30T00:51:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.008+0000 7f3734db7700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37300a71f0 con 0x7f373013ac20 2023-09-30T00:51:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.008+0000 7f3734db7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.008+0000 7f3734db7700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3730141e40 con 0x7f37300b1830 2023-09-30T00:51:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.008+0000 7f3734db7700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.008+0000 7f3734db7700 10 monclient(hunting): authenticate will time out at 5563.309123s 2023-09-30T00:51:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.008+0000 7f372e59c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f37300b1460 0x7f3730141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:47720/0 (socket says 172.21.15.40:47720) 2023-09-30T00:51:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.008+0000 7f372e59c700 1 -- 172.21.15.40:0/3331056071 learned_addr learned my addr 172.21.15.40:0/3331056071 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:51:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.008+0000 7f372dd9b700 1 --2- 172.21.15.40:0/3331056071 >> v2:172.21.15.135:3300/0 conn(0x7f37300b1830 0x7f37300a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.009+0000 7f372dd9b700 10 monclient(hunting): get_auth_request con 0x7f37300b1830 auth_method 0 2023-09-30T00:51:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.009+0000 7f372dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.009+0000 7f372dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.009+0000 7f372dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:42.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.010+0000 7f372dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:42.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.010+0000 7f372dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:42.014 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.010+0000 7f372dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:42.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.012+0000 7f372d59a700 1 -- 172.21.15.40:0/3331056071 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661196180 0 0) 0x7f37300a5b20 con 0x7f37300b1460 2023-09-30T00:51:42.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.012+0000 7f372d59a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:42.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.012+0000 7f372d59a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:42.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.012+0000 7f372d59a700 10 monclient(hunting): my global_id is 4665 2023-09-30T00:51:42.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.012+0000 7f372d59a700 1 -- 172.21.15.40:0/3331056071 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3718003570 con 0x7f37300b1460 2023-09-30T00:51:42.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.014+0000 7f372dd9b700 10 monclient(hunting): handle_auth_done global_id 4723 payload 274 2023-09-30T00:51:42.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.014+0000 7f372dd9b700 1 -- 172.21.15.40:0/3331056071 >> v1:172.21.15.135:6789/0 conn(0x7f373013ac20 legacy=0x7f373013b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:42.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.014+0000 7f372dd9b700 1 -- 172.21.15.40:0/3331056071 >> v1:172.21.15.40:6789/0 conn(0x7f37300b1460 legacy=0x7f3730141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:42.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.014+0000 7f372dd9b700 10 monclient: _finish_hunting 0 2023-09-30T00:51:42.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.014+0000 7f372dd9b700 1 monclient: found mon.noname-c 2023-09-30T00:51:42.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.014+0000 7f372dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:42.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.014+0000 7f372dd9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:51:42.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.014+0000 7f372dd9b700 1 -- 172.21.15.40:0/3331056071 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37300a74d0 con 0x7f37300b1830 2023-09-30T00:51:42.017 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.017+0000 7f372dd9b700 1 --2- 172.21.15.40:0/3331056071 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37300b1830 0x7f37300a5fb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3724009910 tx=0x7f37240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2154e0ab118117eb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f372d59a700 1 -- 172.21.15.40:0/3331056071 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724011630 con 0x7f37300b1830 2023-09-30T00:51:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f372d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f372d59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:51:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f372d59a700 10 monclient: dump: 2023-09-30T00:51:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:42.026 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:42.026 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:42.026 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:42.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f372d59a700 10 monclient: _finish_auth 0 2023-09-30T00:51:42.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f372d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:42.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f372d59a700 1 -- 172.21.15.40:0/3331056071 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3724011790 con 0x7f37300b1830 2023-09-30T00:51:42.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f372d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:42.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f372d59a700 1 -- 172.21.15.40:0/3331056071 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724011b50 con 0x7f37300b1830 2023-09-30T00:51:42.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f372d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:42.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f372d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:42.026 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f372d59a700 10 monclient: dump: 2023-09-30T00:51:42.026 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:42.027 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f3734db7700 5 monclient: authenticate success, global_id 4723 2023-09-30T00:51:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f3734db7700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.024+0000 7f3734db7700 4 set_mon_vals no callback set 2023-09-30T00:51:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.025+0000 7f3734db7700 10 monclient: shutdown 2023-09-30T00:51:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.025+0000 7f3734db7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.025+0000 7f3734db7700 1 -- 172.21.15.40:0/3331056071 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37300b1830 msgr2=0x7f37300a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.025+0000 7f3734db7700 1 --2- 172.21.15.40:0/3331056071 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37300b1830 0x7f37300a5fb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3724009910 tx=0x7f37240091a0 comp rx=0 tx=0).stop 2023-09-30T00:51:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.025+0000 7f3734db7700 1 -- 172.21.15.40:0/3331056071 shutdown_connections 2023-09-30T00:51:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.025+0000 7f3734db7700 1 --2- 172.21.15.40:0/3331056071 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37300b1830 0x7f37300a5fb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.025+0000 7f3734db7700 1 -- 172.21.15.40:0/3331056071 >> 172.21.15.40:0/3331056071 conn(0x7f37300631f0 msgr2=0x7f37300635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.025+0000 7f3734db7700 1 -- 172.21.15.40:0/3331056071 shutdown_connections 2023-09-30T00:51:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.025+0000 7f3734db7700 1 -- 172.21.15.40:0/3331056071 wait complete. 2023-09-30T00:51:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.026+0000 7f3734db7700 10 monclient: build_initial_monmap 2023-09-30T00:51:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.026+0000 7f3734db7700 10 monclient: monmap: 2023-09-30T00:51:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:42.033 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.026+0000 7f3734db7700 1 librados: starting msgr at 2023-09-30T00:51:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.026+0000 7f3734db7700 1 librados: starting objecter 2023-09-30T00:51:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.026+0000 7f3734db7700 1 Processor -- start 2023-09-30T00:51:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.026+0000 7f3734db7700 1 -- start start 2023-09-30T00:51:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.026+0000 7f3734db7700 1 librados: setting wanted keys 2023-09-30T00:51:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.026+0000 7f3734db7700 1 librados: calling monclient init 2023-09-30T00:51:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.026+0000 7f3734db7700 10 monclient: init 2023-09-30T00:51:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:51:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37300b1460 0x7f373013e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 10 monclient(hunting): picked mon.noname-a con 0x7f37300b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37300b1830 0x7f37300a7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 10 monclient(hunting): picked mon.noname-c con 0x7f37300b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f373013ac20 0x7f37300a83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 10 monclient(hunting): picked mon.noname-b con 0x7f373013ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37300ab520 con 0x7f37300b1460 2023-09-30T00:51:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37300a9480 con 0x7f373013ac20 2023-09-30T00:51:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37300a9640 con 0x7f37300b1830 2023-09-30T00:51:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f3734db7700 10 monclient(hunting): authenticate will time out at 5563.328176s 2023-09-30T00:51:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f372e59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37300b1460 0x7f373013e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f372dd9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37300b1830 0x7f37300a7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.027+0000 7f372ed9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f373013ac20 0x7f37300a83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372ed9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f373013ac20 0x7f37300a83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:58276/0 (socket says 172.21.15.40:58276) 2023-09-30T00:51:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372ed9d700 1 -- 172.21.15.40:0/1477450129 learned_addr learned my addr 172.21.15.40:0/1477450129 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372ed9d700 10 monclient(hunting): get_auth_request con 0x7f373013ac20 auth_method 0 2023-09-30T00:51:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372dd9b700 10 monclient(hunting): get_auth_request con 0x7f37300b1830 auth_method 0 2023-09-30T00:51:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372e59c700 10 monclient(hunting): get_auth_request con 0x7f37300b1460 auth_method 0 2023-09-30T00:51:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.028+0000 7f372ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.029+0000 7f372e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:42.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.029+0000 7f372e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.029+0000 7f372e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.029+0000 7f372dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.029+0000 7f372dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.029+0000 7f372dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.032+0000 7f372ed9d700 10 monclient(hunting): handle_auth_done global_id 4726 payload 824 2023-09-30T00:51:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.032+0000 7f372ed9d700 1 -- 172.21.15.40:0/1477450129 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37300b1830 msgr2=0x7f37300a7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.032+0000 7f372ed9d700 1 --2- 172.21.15.40:0/1477450129 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37300b1830 0x7f37300a7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.032+0000 7f372ed9d700 1 -- 172.21.15.40:0/1477450129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37300b1460 msgr2=0x7f373013e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.032+0000 7f372ed9d700 1 --2- 172.21.15.40:0/1477450129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37300b1460 0x7f373013e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.032+0000 7f372ed9d700 10 monclient: _finish_hunting 0 2023-09-30T00:51:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.032+0000 7f372ed9d700 1 monclient: found mon.noname-b 2023-09-30T00:51:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.032+0000 7f372ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.032+0000 7f372ed9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:51:42.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.032+0000 7f372ed9d700 1 -- 172.21.15.40:0/1477450129 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3724009420 con 0x7f373013ac20 2023-09-30T00:51:42.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.035+0000 7f372ed9d700 1 --2- 172.21.15.40:0/1477450129 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f373013ac20 0x7f37300a83e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f372800bc30 tx=0x7f3728009820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:42.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.035+0000 7f371e7fc700 10 client.?.objecter ms_handle_connect 0x7f373013ac20 2023-09-30T00:51:42.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.035+0000 7f371e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f371e7fc700 1 -- 172.21.15.40:0/1477450129 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372800c840 con 0x7f373013ac20 2023-09-30T00:51:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f371e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f371e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f371e7fc700 10 monclient: dump: 2023-09-30T00:51:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:42.041 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f371e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:51:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f371e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f371e7fc700 1 -- 172.21.15.40:0/1477450129 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372800c9e0 con 0x7f373013ac20 2023-09-30T00:51:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f3734db7700 5 monclient: authenticate success, global_id 4726 2023-09-30T00:51:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f3734db7700 10 monclient: _renew_subs 2023-09-30T00:51:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f3734db7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f3734db7700 1 -- 172.21.15.40:0/1477450129 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37300a9800 con 0x7f373013ac20 2023-09-30T00:51:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f371e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f371e7fc700 1 -- 172.21.15.40:0/1477450129 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372800cda0 con 0x7f373013ac20 2023-09-30T00:51:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f371e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f371e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f372cd99700 4 set_mon_vals no callback set 2023-09-30T00:51:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f371e7fc700 10 monclient: dump: 2023-09-30T00:51:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:42.042 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:42.043 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:42.043 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:42.043 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:42.043 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:42.043 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:42.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f3734db7700 10 client.4726.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:42.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f3734db7700 10 monclient: _renew_subs 2023-09-30T00:51:42.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f3734db7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:42.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f3734db7700 1 -- 172.21.15.40:0/1477450129 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37300a9d40 con 0x7f373013ac20 2023-09-30T00:51:42.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.039+0000 7f3734db7700 1 librados: init done 2023-09-30T00:51:42.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.040+0000 7f3734db7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:42.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.040+0000 7f3734db7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:42.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.040+0000 7f3734db7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:42.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.040+0000 7f3734db7700 1 -- 172.21.15.40:0/1477450129 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3730064a00 con 0x7f373013ac20 2023-09-30T00:51:42.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.059+0000 7f371e7fc700 1 -- 172.21.15.40:0/1477450129 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3728014620 con 0x7f373013ac20 2023-09-30T00:51:42.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.060+0000 7f371e7fc700 1 --2- 172.21.15.40:0/1477450129 >> v2:172.21.15.135:6800/116426 conn(0x7f3710068960 0x7f371006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:42.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.060+0000 7f372e59c700 1 --2- 172.21.15.40:0/1477450129 >> v2:172.21.15.135:6800/116426 conn(0x7f3710068960 0x7f371006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:42.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.060+0000 7f372e59c700 10 monclient: get_auth_request con 0x7f3710068960 auth_method 0 2023-09-30T00:51:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.060+0000 7f372e59c700 1 --2- 172.21.15.40:0/1477450129 >> v2:172.21.15.135:6800/116426 conn(0x7f3710068960 0x7f371006ae10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3720007fd0 tx=0x7f3720009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:42.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.060+0000 7f371e7fc700 10 client.4726.objecter ms_handle_connect 0x7f3710068960 2023-09-30T00:51:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.229+0000 7f371e7fc700 1 -- 172.21.15.40:0/1477450129 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3728056330 con 0x7f373013ac20 2023-09-30T00:51:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.229+0000 7f371e7fc700 10 client.4726.objecter ms_dispatch 0x7f3730001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.229+0000 7f371e7fc700 3 client.4726.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.229+0000 7f371e7fc700 3 client.4726.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:42.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.229+0000 7f371e7fc700 20 client.4726.objecter dump_active .. 0 homeless 2023-09-30T00:51:42.303 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.303+0000 7f371e7fc700 1 -- 172.21.15.40:0/1477450129 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3728051f70 con 0x7f373013ac20 2023-09-30T00:51:42.304 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.303+0000 7f371e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:42.304 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.303+0000 7f371e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:42.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.589+0000 7f3734db7700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:42.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.589+0000 7f3734db7700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:42.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.589+0000 7f3734db7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:42.590 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.589+0000 7f3734db7700 1 -- 172.21.15.40:0/1477450129 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f37300b42c0 con 0x7f373013ac20 2023-09-30T00:51:42.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.605+0000 7f371e7fc700 1 -- 172.21.15.40:0/1477450129 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3728022080 con 0x7f373013ac20 2023-09-30T00:51:42.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.605+0000 7f371e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:42.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.605+0000 7f371e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:42.606 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:42.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.607+0000 7f3734db7700 10 librados: watch_flush enter 2023-09-30T00:51:42.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.607+0000 7f3734db7700 10 librados: watch_flush exit 2023-09-30T00:51:42.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.607+0000 7f3734db7700 20 client.4726.objecter shutdown clearing up homeless session... 2023-09-30T00:51:42.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.607+0000 7f3734db7700 10 client.4726.objecter successfully canceled tick 2023-09-30T00:51:42.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.607+0000 7f3734db7700 1 -- 172.21.15.40:0/1477450129 >> v2:172.21.15.135:6800/116426 conn(0x7f3710068960 msgr2=0x7f371006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:42.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.607+0000 7f3734db7700 1 --2- 172.21.15.40:0/1477450129 >> v2:172.21.15.135:6800/116426 conn(0x7f3710068960 0x7f371006ae10 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3720007fd0 tx=0x7f3720009040 comp rx=0 tx=0).stop 2023-09-30T00:51:42.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.607+0000 7f3734db7700 10 monclient: shutdown 2023-09-30T00:51:42.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.607+0000 7f3734db7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:42.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.607+0000 7f3734db7700 1 -- 172.21.15.40:0/1477450129 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f373013ac20 msgr2=0x7f37300a83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:42.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.607+0000 7f3734db7700 1 --2- 172.21.15.40:0/1477450129 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f373013ac20 0x7f37300a83e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f372800bc30 tx=0x7f3728009820 comp rx=0 tx=0).stop 2023-09-30T00:51:42.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.608+0000 7f3734db7700 1 -- 172.21.15.40:0/1477450129 shutdown_connections 2023-09-30T00:51:42.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.608+0000 7f3734db7700 1 --2- 172.21.15.40:0/1477450129 >> v2:172.21.15.135:6800/116426 conn(0x7f3710068960 0x7f371006ae10 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:42.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.608+0000 7f3734db7700 1 --2- 172.21.15.40:0/1477450129 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f373013ac20 0x7f37300a83e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:42.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.608+0000 7f3734db7700 1 --2- 172.21.15.40:0/1477450129 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37300b1460 0x7f373013e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:42.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.608+0000 7f3734db7700 1 --2- 172.21.15.40:0/1477450129 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37300b1830 0x7f37300a7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:42.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.608+0000 7f3734db7700 1 -- 172.21.15.40:0/1477450129 >> 172.21.15.40:0/1477450129 conn(0x7f37300631f0 msgr2=0x7f37300635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:42.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.608+0000 7f3734db7700 1 -- 172.21.15.40:0/1477450129 shutdown_connections 2023-09-30T00:51:42.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.608+0000 7f3734db7700 1 -- 172.21.15.40:0/1477450129 wait complete. 2023-09-30T00:51:42.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:42.608+0000 7f3734db7700 1 librados: shutdown 2023-09-30T00:51:45.623 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:45.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.765+0000 7f9cc32bf700 10 monclient: get_monmap_and_config 2023-09-30T00:51:45.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.765+0000 7f9cc32bf700 10 monclient: build_initial_monmap 2023-09-30T00:51:45.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.765+0000 7f9cc32bf700 10 monclient: monmap: 2023-09-30T00:51:45.766 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:45.766 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:45.766 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:45.766128+0000 2023-09-30T00:51:45.766 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:45.766128+0000 2023-09-30T00:51:45.766 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:45.766 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:45.766 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:45.766 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:45.766 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:45.766 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:45.767 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:45.767 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:45.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.765+0000 7f9cc32bf700 1 Processor -- start 2023-09-30T00:51:45.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 1 -- start start 2023-09-30T00:51:45.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 10 monclient: init 2023-09-30T00:51:45.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:45.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-09-30T00:51:45.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9cbc06e490 addr v1:172.21.15.135:6789/0 2023-09-30T00:51:45.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9cbc06e860 0x7f9cbc13c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:45.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 10 monclient(hunting): picked mon.noname-a con 0x7f9cbc06e860 addr v2:172.21.15.40:3300/0 2023-09-30T00:51:45.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9cbc139bf0 0x7f9cbc13cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:45.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 10 monclient(hunting): picked mon.noname-c con 0x7f9cbc139bf0 addr v2:172.21.15.135:3300/0 2023-09-30T00:51:45.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cbc0aa6b0 con 0x7f9cbc06e490 2023-09-30T00:51:45.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:45.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cbc06e120 con 0x7f9cbc06e860 2023-09-30T00:51:45.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:45.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cbc06f750 con 0x7f9cbc139bf0 2023-09-30T00:51:45.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:45.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.766+0000 7f9cc32bf700 10 monclient(hunting): authenticate will time out at 5567.067810s 2023-09-30T00:51:45.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.767+0000 7f9cc185c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9cbc139bf0 0x7f9cbc13cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:45.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.768+0000 7f9cc185c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9cbc139bf0 0x7f9cbc13cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:58292/0 (socket says 172.21.15.40:58292) 2023-09-30T00:51:45.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.768+0000 7f9cc185c700 1 -- 172.21.15.40:0/1059666237 learned_addr learned my addr 172.21.15.40:0/1059666237 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:45.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.768+0000 7f9cc185c700 10 monclient(hunting): get_auth_request con 0x7f9cbc139bf0 auth_method 0 2023-09-30T00:51:45.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.768+0000 7f9cc185c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:45.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.768+0000 7f9cc185c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:45.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.768+0000 7f9cc185c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:45.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.769+0000 7f9cc185c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:45.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.769+0000 7f9cc185c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:45.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.769+0000 7f9cc185c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:45.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.771+0000 7f9cc185c700 10 monclient(hunting): handle_auth_done global_id 4729 payload 274 2023-09-30T00:51:45.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.771+0000 7f9cc185c700 1 -- 172.21.15.40:0/1059666237 >> v2:172.21.15.40:3300/0 conn(0x7f9cbc06e860 msgr2=0x7f9cbc13c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:45.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.771+0000 7f9cc185c700 1 --2- 172.21.15.40:0/1059666237 >> v2:172.21.15.40:3300/0 conn(0x7f9cbc06e860 0x7f9cbc13c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:45.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.771+0000 7f9cc185c700 1 -- 172.21.15.40:0/1059666237 >> v1:172.21.15.135:6789/0 conn(0x7f9cbc06e490 legacy=0x7f9cbc1394e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:45.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.771+0000 7f9cc185c700 10 monclient: _finish_hunting 0 2023-09-30T00:51:45.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.771+0000 7f9cc185c700 1 monclient: found mon.noname-c 2023-09-30T00:51:45.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.772+0000 7f9cc185c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:45.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.772+0000 7f9cc185c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:51:45.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.772+0000 7f9cc185c700 1 -- 172.21.15.40:0/1059666237 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cbc0aaff0 con 0x7f9cbc139bf0 2023-09-30T00:51:45.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.774+0000 7f9cc185c700 1 --2- 172.21.15.40:0/1059666237 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cbc139bf0 0x7f9cbc13cab0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc06e090 tx=0x7f9cac009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=974ec5910be2814 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:45.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cbbfff700 1 -- 172.21.15.40:0/1059666237 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac00e070 con 0x7f9cbc139bf0 2023-09-30T00:51:45.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cbbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:45.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cbbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:51:45.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cbbfff700 10 monclient: dump: 2023-09-30T00:51:45.782 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:45.782 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:45.782 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:45.782 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:45.782 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:45.782 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:45.782 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:45.783 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:45.783 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:45.783 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:45.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cbbfff700 10 monclient: _finish_auth 0 2023-09-30T00:51:45.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cbbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:45.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cbbfff700 1 -- 172.21.15.40:0/1059666237 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cac00ba10 con 0x7f9cbc139bf0 2023-09-30T00:51:45.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cbbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:45.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cbbfff700 1 -- 172.21.15.40:0/1059666237 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac0092e0 con 0x7f9cbc139bf0 2023-09-30T00:51:45.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cbbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:45.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cbbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:45.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cbbfff700 10 monclient: dump: 2023-09-30T00:51:45.784 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:45.784 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:45.784 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:45.784 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:45.784 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:45.784 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:45.784 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:45.784 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:45.784 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:45.784 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:45.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cc32bf700 5 monclient: authenticate success, global_id 4729 2023-09-30T00:51:45.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cc32bf700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:45.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.781+0000 7f9cc32bf700 4 set_mon_vals no callback set 2023-09-30T00:51:45.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.782+0000 7f9cc32bf700 10 monclient: shutdown 2023-09-30T00:51:45.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.782+0000 7f9cc32bf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:45.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.782+0000 7f9cc32bf700 1 -- 172.21.15.40:0/1059666237 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cbc139bf0 msgr2=0x7f9cbc13cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:45.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.782+0000 7f9cc32bf700 1 --2- 172.21.15.40:0/1059666237 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cbc139bf0 0x7f9cbc13cab0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc06e090 tx=0x7f9cac009840 comp rx=0 tx=0).stop 2023-09-30T00:51:45.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.782+0000 7f9cc32bf700 1 -- 172.21.15.40:0/1059666237 shutdown_connections 2023-09-30T00:51:45.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.782+0000 7f9cc32bf700 1 --2- 172.21.15.40:0/1059666237 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cbc139bf0 0x7f9cbc13cab0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:45.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.782+0000 7f9cc32bf700 1 --2- 172.21.15.40:0/1059666237 >> v2:172.21.15.40:3300/0 conn(0x7f9cbc06e860 0x7f9cbc13c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:45.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.782+0000 7f9cc32bf700 1 -- 172.21.15.40:0/1059666237 >> 172.21.15.40:0/1059666237 conn(0x7f9cbc0680a0 msgr2=0x7f9cbc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:45.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.782+0000 7f9cc32bf700 1 -- 172.21.15.40:0/1059666237 shutdown_connections 2023-09-30T00:51:45.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.782+0000 7f9cc32bf700 1 -- 172.21.15.40:0/1059666237 wait complete. 2023-09-30T00:51:45.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.782+0000 7f9cc32bf700 10 monclient: build_initial_monmap 2023-09-30T00:51:45.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.782+0000 7f9cc32bf700 10 monclient: monmap: 2023-09-30T00:51:45.788 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:45.788 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:45.788 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:45.788 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:45.788 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:45.788 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:45.788 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:45.788 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:45.789 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:45.789 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:45.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.783+0000 7f9cc32bf700 1 librados: starting msgr at 2023-09-30T00:51:45.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.783+0000 7f9cc32bf700 1 librados: starting objecter 2023-09-30T00:51:45.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.783+0000 7f9cc32bf700 1 Processor -- start 2023-09-30T00:51:45.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.783+0000 7f9cc32bf700 1 -- start start 2023-09-30T00:51:45.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.783+0000 7f9cc32bf700 1 librados: setting wanted keys 2023-09-30T00:51:45.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.783+0000 7f9cc32bf700 1 librados: calling monclient init 2023-09-30T00:51:45.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.783+0000 7f9cc32bf700 10 monclient: init 2023-09-30T00:51:45.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:45.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:51:45.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cbc06e490 0x7f9cbc13f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:45.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 10 monclient(hunting): picked mon.noname-a con 0x7f9cbc06e490 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:45.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cbc06e860 0x7f9cbc14f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:45.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 10 monclient(hunting): picked mon.noname-b con 0x7f9cbc06e860 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:45.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9cbc139bf0 0x7f9cbc14b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:45.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 10 monclient(hunting): picked mon.noname-c con 0x7f9cbc139bf0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:45.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:45.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc0a6e90 con 0x7f9cbc06e490 2023-09-30T00:51:45.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:45.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc0a9210 con 0x7f9cbc06e860 2023-09-30T00:51:45.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:45.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc14f590 con 0x7f9cbc139bf0 2023-09-30T00:51:45.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:45.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.784+0000 7f9cc32bf700 10 monclient(hunting): authenticate will time out at 5567.085281s 2023-09-30T00:51:45.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.785+0000 7f9cc085a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cbc06e860 0x7f9cbc14f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:45.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.786+0000 7f9cc085a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cbc06e860 0x7f9cbc14f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:58304/0 (socket says 172.21.15.40:58304) 2023-09-30T00:51:45.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.786+0000 7f9cc085a700 1 -- 172.21.15.40:0/284910390 learned_addr learned my addr 172.21.15.40:0/284910390 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:45.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.786+0000 7f9cc085a700 10 monclient(hunting): get_auth_request con 0x7f9cbc06e860 auth_method 0 2023-09-30T00:51:45.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.786+0000 7f9cc085a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:45.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.786+0000 7f9cc085a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:45.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.786+0000 7f9cc085a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:45.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.786+0000 7f9cc085a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:45.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.786+0000 7f9cc085a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:45.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.786+0000 7f9cc085a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:45.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.790+0000 7f9cc085a700 10 monclient(hunting): handle_auth_done global_id 4732 payload 824 2023-09-30T00:51:45.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.790+0000 7f9cc085a700 1 -- 172.21.15.40:0/284910390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9cbc139bf0 msgr2=0x7f9cbc14b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:45.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.790+0000 7f9cc085a700 1 --2- 172.21.15.40:0/284910390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9cbc139bf0 0x7f9cbc14b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:45.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.790+0000 7f9cc085a700 1 -- 172.21.15.40:0/284910390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cbc06e490 msgr2=0x7f9cbc13f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:45.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.790+0000 7f9cc085a700 1 --2- 172.21.15.40:0/284910390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cbc06e490 0x7f9cbc13f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:45.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.790+0000 7f9cc085a700 10 monclient: _finish_hunting 0 2023-09-30T00:51:45.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.790+0000 7f9cc085a700 1 monclient: found mon.noname-b 2023-09-30T00:51:45.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.790+0000 7f9cc085a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:45.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.790+0000 7f9cc085a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:51:45.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.790+0000 7f9cc085a700 1 -- 172.21.15.40:0/284910390 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac009ae0 con 0x7f9cbc06e860 2023-09-30T00:51:45.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.793+0000 7f9cc085a700 1 --2- 172.21.15.40:0/284910390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cbc06e860 0x7f9cbc14f050 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000b810 tx=0x7f9cb0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:45.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.793+0000 7f9cb97fa700 10 client.?.objecter ms_handle_connect 0x7f9cbc06e860 2023-09-30T00:51:45.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.793+0000 7f9cb97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:45.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cb97fa700 1 -- 172.21.15.40:0/284910390 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb000f5f0 con 0x7f9cbc06e860 2023-09-30T00:51:45.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cb97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:45.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cb97fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:45.846 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cb97fa700 10 monclient: dump: 2023-09-30T00:51:45.846 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:45.846 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:45.846 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:45.847 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:45.847 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:45.847 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:45.847 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:45.847 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:45.847 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:45.847 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:45.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cb97fa700 10 monclient: _finish_auth 0 2023-09-30T00:51:45.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cb97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:45.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cc32bf700 5 monclient: authenticate success, global_id 4732 2023-09-30T00:51:45.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cc32bf700 10 monclient: _renew_subs 2023-09-30T00:51:45.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cc32bf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:45.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cc32bf700 1 -- 172.21.15.40:0/284910390 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cbc14f810 con 0x7f9cbc06e860 2023-09-30T00:51:45.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cc32bf700 10 client.4732.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:45.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cc32bf700 10 monclient: _renew_subs 2023-09-30T00:51:45.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cc32bf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:45.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cc32bf700 1 -- 172.21.15.40:0/284910390 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cbc14c150 con 0x7f9cbc06e860 2023-09-30T00:51:45.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.845+0000 7f9cc32bf700 1 librados: init done 2023-09-30T00:51:45.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.846+0000 7f9cc32bf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:45.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.846+0000 7f9cc32bf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:45.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.846+0000 7f9cc32bf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:45.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:45.846+0000 7f9cc32bf700 1 -- 172.21.15.40:0/284910390 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cbc072b90 con 0x7f9cbc06e860 2023-09-30T00:51:46.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.143+0000 7f9cb97fa700 1 -- 172.21.15.40:0/284910390 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb000f960 con 0x7f9cbc06e860 2023-09-30T00:51:46.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.143+0000 7f9cb97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:46.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.143+0000 7f9cb97fa700 1 -- 172.21.15.40:0/284910390 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb000fad0 con 0x7f9cbc06e860 2023-09-30T00:51:46.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.143+0000 7f9cb97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:46.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.143+0000 7f9cb97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.143+0000 7f9cb97fa700 10 monclient: dump: 2023-09-30T00:51:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:46.145 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:46.145 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:46.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.143+0000 7f9cbb7fe700 4 set_mon_vals no callback set 2023-09-30T00:51:46.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.143+0000 7f9cb97fa700 1 -- 172.21.15.40:0/284910390 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9cb000fcb0 con 0x7f9cbc06e860 2023-09-30T00:51:46.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.144+0000 7f9cb97fa700 1 --2- 172.21.15.40:0/284910390 >> v2:172.21.15.135:6800/116426 conn(0x7f9ca8068890 0x7f9ca806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:46.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.144+0000 7f9cc105b700 1 --2- 172.21.15.40:0/284910390 >> v2:172.21.15.135:6800/116426 conn(0x7f9ca8068890 0x7f9ca806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:46.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.144+0000 7f9cc105b700 10 monclient: get_auth_request con 0x7f9ca8068890 auth_method 0 2023-09-30T00:51:46.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.145+0000 7f9cc105b700 1 --2- 172.21.15.40:0/284910390 >> v2:172.21.15.135:6800/116426 conn(0x7f9ca8068890 0x7f9ca806ad40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4009740 tx=0x7f9cb4006c60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:46.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.145+0000 7f9cb97fa700 10 client.4732.objecter ms_handle_connect 0x7f9ca8068890 2023-09-30T00:51:46.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.148+0000 7f9cb97fa700 1 -- 172.21.15.40:0/284910390 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9cb0056410 con 0x7f9cbc06e860 2023-09-30T00:51:46.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.148+0000 7f9cb97fa700 10 client.4732.objecter ms_dispatch 0x7f9cbc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:46.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.148+0000 7f9cb97fa700 3 client.4732.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:46.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.148+0000 7f9cb97fa700 3 client.4732.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:46.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.148+0000 7f9cb97fa700 20 client.4732.objecter dump_active .. 0 homeless 2023-09-30T00:51:46.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.163+0000 7f9cb97fa700 1 -- 172.21.15.40:0/284910390 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9cb0052050 con 0x7f9cbc06e860 2023-09-30T00:51:46.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.163+0000 7f9cb97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:46.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.163+0000 7f9cb97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:46.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.419+0000 7f9cc32bf700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.419+0000 7f9cc32bf700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.419+0000 7f9cc32bf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:46.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.419+0000 7f9cc32bf700 1 -- 172.21.15.40:0/284910390 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9cbc0aac80 con 0x7f9cbc06e860 2023-09-30T00:51:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.462+0000 7f9cb97fa700 1 -- 172.21.15.40:0/284910390 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9cb0055670 con 0x7f9cbc06e860 2023-09-30T00:51:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.462+0000 7f9cb97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:46.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.462+0000 7f9cb97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:46.463 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:46.463 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:46.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 10 librados: watch_flush enter 2023-09-30T00:51:46.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 10 librados: watch_flush exit 2023-09-30T00:51:46.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 20 client.4732.objecter shutdown clearing up homeless session... 2023-09-30T00:51:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 10 client.4732.objecter successfully canceled tick 2023-09-30T00:51:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 1 -- 172.21.15.40:0/284910390 >> v2:172.21.15.135:6800/116426 conn(0x7f9ca8068890 msgr2=0x7f9ca806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 1 --2- 172.21.15.40:0/284910390 >> v2:172.21.15.135:6800/116426 conn(0x7f9ca8068890 0x7f9ca806ad40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4009740 tx=0x7f9cb4006c60 comp rx=0 tx=0).stop 2023-09-30T00:51:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 10 monclient: shutdown 2023-09-30T00:51:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 1 -- 172.21.15.40:0/284910390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cbc06e860 msgr2=0x7f9cbc14f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 1 --2- 172.21.15.40:0/284910390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cbc06e860 0x7f9cbc14f050 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000b810 tx=0x7f9cb0009250 comp rx=0 tx=0).stop 2023-09-30T00:51:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 1 -- 172.21.15.40:0/284910390 shutdown_connections 2023-09-30T00:51:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 1 --2- 172.21.15.40:0/284910390 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9cbc139bf0 0x7f9cbc14b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 1 --2- 172.21.15.40:0/284910390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9cbc06e490 0x7f9cbc13f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 1 --2- 172.21.15.40:0/284910390 >> v2:172.21.15.135:6800/116426 conn(0x7f9ca8068890 0x7f9ca806ad40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:46.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 1 --2- 172.21.15.40:0/284910390 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9cbc06e860 0x7f9cbc14f050 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:46.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.464+0000 7f9cc32bf700 1 -- 172.21.15.40:0/284910390 >> 172.21.15.40:0/284910390 conn(0x7f9cbc0680a0 msgr2=0x7f9cbc06d540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:46.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.465+0000 7f9cc32bf700 1 -- 172.21.15.40:0/284910390 shutdown_connections 2023-09-30T00:51:46.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.465+0000 7f9cc32bf700 1 -- 172.21.15.40:0/284910390 wait complete. 2023-09-30T00:51:46.467 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:46.465+0000 7f9cc32bf700 1 librados: shutdown 2023-09-30T00:51:49.482 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.620+0000 7fd589fe2700 10 monclient: get_monmap_and_config 2023-09-30T00:51:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.620+0000 7fd589fe2700 10 monclient: build_initial_monmap 2023-09-30T00:51:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.620+0000 7fd589fe2700 10 monclient: monmap: 2023-09-30T00:51:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:49.621617+0000 2023-09-30T00:51:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:49.621617+0000 2023-09-30T00:51:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:49.622 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:49.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.621+0000 7fd589fe2700 1 Processor -- start 2023-09-30T00:51:49.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.621+0000 7fd589fe2700 1 -- start start 2023-09-30T00:51:49.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.621+0000 7fd589fe2700 10 monclient: init 2023-09-30T00:51:49.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd589fe2700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:49.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd589fe2700 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-09-30T00:51:49.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd589fe2700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd5840b1460 0x7fd5840b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:49.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd589fe2700 10 monclient(hunting): picked mon.noname-c con 0x7fd5840b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:51:49.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd589fe2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5840a5fb0 0x7fd584141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:49.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd589fe2700 10 monclient(hunting): picked mon.noname-b con 0x7fd5840a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:49.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd589fe2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5840af410 addr v1:172.21.15.135:6789/0 2023-09-30T00:51:49.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd589fe2700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5840a71f0 con 0x7fd5840af410 2023-09-30T00:51:49.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd589fe2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:49.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd589fe2700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5840a74d0 con 0x7fd5840b1460 2023-09-30T00:51:49.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd589fe2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:49.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd589fe2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5840a63c0 con 0x7fd5840a5fb0 2023-09-30T00:51:49.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd589fe2700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:49.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd589fe2700 10 monclient(hunting): authenticate will time out at 5570.923182s 2023-09-30T00:51:49.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.622+0000 7fd582ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5840a5fb0 0x7fd584141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:49.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.623+0000 7fd5837fe700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd5840b1460 0x7fd5840b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:49.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.623+0000 7fd582ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5840a5fb0 0x7fd584141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49354/0 (socket says 172.21.15.40:49354) 2023-09-30T00:51:49.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.623+0000 7fd582ffd700 1 -- 172.21.15.40:0/783318258 learned_addr learned my addr 172.21.15.40:0/783318258 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:49.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.623+0000 7fd582ffd700 10 monclient(hunting): get_auth_request con 0x7fd5840a5fb0 auth_method 0 2023-09-30T00:51:49.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.623+0000 7fd582ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:49.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.623+0000 7fd582ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:49.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.623+0000 7fd582ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:49.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.623+0000 7fd5837fe700 10 monclient(hunting): get_auth_request con 0x7fd5840b1460 auth_method 0 2023-09-30T00:51:49.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.623+0000 7fd5837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:49.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.623+0000 7fd5837fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.623+0000 7fd5837fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.623+0000 7fd582ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.623+0000 7fd582ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.624+0000 7fd582ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.625+0000 7fd5837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.625+0000 7fd5837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.625+0000 7fd5837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.626+0000 7fd582ffd700 10 monclient(hunting): handle_auth_done global_id 4592 payload 274 2023-09-30T00:51:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.626+0000 7fd582ffd700 1 -- 172.21.15.40:0/783318258 >> v2:172.21.15.135:3300/0 conn(0x7fd5840b1460 msgr2=0x7fd5840b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.626+0000 7fd582ffd700 1 --2- 172.21.15.40:0/783318258 >> v2:172.21.15.135:3300/0 conn(0x7fd5840b1460 0x7fd5840b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.626+0000 7fd582ffd700 1 -- 172.21.15.40:0/783318258 >> v1:172.21.15.135:6789/0 conn(0x7fd5840af410 legacy=0x7fd5840af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.626+0000 7fd582ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:51:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.626+0000 7fd582ffd700 1 monclient: found mon.noname-b 2023-09-30T00:51:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.626+0000 7fd582ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.626+0000 7fd582ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:51:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.626+0000 7fd582ffd700 1 -- 172.21.15.40:0/783318258 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5840a56d0 con 0x7fd5840a5fb0 2023-09-30T00:51:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.628+0000 7fd582ffd700 1 --2- 172.21.15.40:0/783318258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5840a5fb0 0x7fd584141730 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd56c0099b0 tx=0x7fd56c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7423555826e472cf server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.632+0000 7fd5827fc700 1 -- 172.21.15.40:0/783318258 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c009ea0 con 0x7fd5840a5fb0 2023-09-30T00:51:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.632+0000 7fd5827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.632+0000 7fd5827fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.632+0000 7fd5827fc700 10 monclient: dump: 2023-09-30T00:51:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:49.634 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:49.634 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:49.634 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:49.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.632+0000 7fd5827fc700 10 monclient: _finish_auth 0 2023-09-30T00:51:49.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.632+0000 7fd5827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:49.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.632+0000 7fd589fe2700 5 monclient: authenticate success, global_id 4592 2023-09-30T00:51:49.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.632+0000 7fd589fe2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:51:49.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.633+0000 7fd5827fc700 1 -- 172.21.15.40:0/783318258 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56c00b860 con 0x7fd5840a5fb0 2023-09-30T00:51:49.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.633+0000 7fd5827fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.633+0000 7fd5827fc700 1 -- 172.21.15.40:0/783318258 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c00bc00 con 0x7fd5840a5fb0 2023-09-30T00:51:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.633+0000 7fd589fe2700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.633+0000 7fd589fe2700 4 set_mon_vals no callback set 2023-09-30T00:51:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.633+0000 7fd5827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.633+0000 7fd5827fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:51:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.633+0000 7fd5827fc700 10 monclient: dump: 2023-09-30T00:51:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:49.638 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.634+0000 7fd589fe2700 10 monclient: shutdown 2023-09-30T00:51:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.634+0000 7fd589fe2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.634+0000 7fd589fe2700 1 -- 172.21.15.40:0/783318258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5840a5fb0 msgr2=0x7fd584141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.634+0000 7fd589fe2700 1 --2- 172.21.15.40:0/783318258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5840a5fb0 0x7fd584141730 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd56c0099b0 tx=0x7fd56c009250 comp rx=0 tx=0).stop 2023-09-30T00:51:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.634+0000 7fd589fe2700 1 -- 172.21.15.40:0/783318258 shutdown_connections 2023-09-30T00:51:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.634+0000 7fd589fe2700 1 --2- 172.21.15.40:0/783318258 >> v2:172.21.15.135:3300/0 conn(0x7fd5840b1460 0x7fd5840b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.634+0000 7fd589fe2700 1 --2- 172.21.15.40:0/783318258 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5840a5fb0 0x7fd584141730 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.634+0000 7fd589fe2700 1 -- 172.21.15.40:0/783318258 >> 172.21.15.40:0/783318258 conn(0x7fd5840631f0 msgr2=0x7fd5840635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:49.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.634+0000 7fd589fe2700 1 -- 172.21.15.40:0/783318258 shutdown_connections 2023-09-30T00:51:49.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.634+0000 7fd589fe2700 1 -- 172.21.15.40:0/783318258 wait complete. 2023-09-30T00:51:49.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.634+0000 7fd589fe2700 10 monclient: build_initial_monmap 2023-09-30T00:51:49.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.634+0000 7fd589fe2700 10 monclient: monmap: 2023-09-30T00:51:49.639 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:49.639 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:49.639 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:49.639 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:49.639 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:49.639 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:49.639 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:49.639 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:49.640 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:49.640 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:49.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.635+0000 7fd589fe2700 1 librados: starting msgr at 2023-09-30T00:51:49.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.635+0000 7fd589fe2700 1 librados: starting objecter 2023-09-30T00:51:49.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.635+0000 7fd589fe2700 1 Processor -- start 2023-09-30T00:51:49.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.635+0000 7fd589fe2700 1 -- start start 2023-09-30T00:51:49.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.635+0000 7fd589fe2700 1 librados: setting wanted keys 2023-09-30T00:51:49.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.635+0000 7fd589fe2700 1 librados: calling monclient init 2023-09-30T00:51:49.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.635+0000 7fd589fe2700 10 monclient: init 2023-09-30T00:51:49.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:49.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:51:49.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5840a5fb0 0x7fd58414f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:49.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 10 monclient(hunting): picked mon.noname-a con 0x7fd5840a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:49.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5840af410 0x7fd5841510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:49.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 10 monclient(hunting): picked mon.noname-b con 0x7fd5840af410 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:49.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5840b1460 0x7fd58414ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:49.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 10 monclient(hunting): picked mon.noname-c con 0x7fd5840b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:49.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:49.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5841515f0 con 0x7fd5840a5fb0 2023-09-30T00:51:49.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:49.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd584151830 con 0x7fd5840af410 2023-09-30T00:51:49.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:49.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5840a79f0 con 0x7fd5840b1460 2023-09-30T00:51:49.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:49.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd589fe2700 10 monclient(hunting): authenticate will time out at 5570.937189s 2023-09-30T00:51:49.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd5837fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5840a5fb0 0x7fd58414f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:49.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.636+0000 7fd582ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5840af410 0x7fd5841510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:49.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd583fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5840b1460 0x7fd58414ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:49.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd582ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5840af410 0x7fd5841510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:58330/0 (socket says 172.21.15.40:58330) 2023-09-30T00:51:49.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd582ffd700 1 -- 172.21.15.40:0/2493849018 learned_addr learned my addr 172.21.15.40:0/2493849018 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:49.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd582ffd700 10 monclient(hunting): get_auth_request con 0x7fd5840af410 auth_method 0 2023-09-30T00:51:49.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd582ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:49.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd582ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:49.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd582ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:49.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd5837fe700 10 monclient(hunting): get_auth_request con 0x7fd5840a5fb0 auth_method 0 2023-09-30T00:51:49.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd5837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:49.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd5837fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:49.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd5837fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:49.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd583fff700 10 monclient(hunting): get_auth_request con 0x7fd5840b1460 auth_method 0 2023-09-30T00:51:49.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd583fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:49.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd583fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:49.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd583fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:49.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd582ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:49.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd582ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:49.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.637+0000 7fd582ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:49.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.638+0000 7fd583fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:49.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.638+0000 7fd583fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:49.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.638+0000 7fd583fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:49.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.638+0000 7fd5837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:49.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.638+0000 7fd5837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:49.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.638+0000 7fd5837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:49.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.641+0000 7fd582ffd700 10 monclient(hunting): handle_auth_done global_id 4741 payload 824 2023-09-30T00:51:49.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.641+0000 7fd582ffd700 1 -- 172.21.15.40:0/2493849018 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5840b1460 msgr2=0x7fd58414ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:49.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.641+0000 7fd582ffd700 1 --2- 172.21.15.40:0/2493849018 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5840b1460 0x7fd58414ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:49.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.641+0000 7fd582ffd700 1 -- 172.21.15.40:0/2493849018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5840a5fb0 msgr2=0x7fd58414f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:49.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.641+0000 7fd582ffd700 1 --2- 172.21.15.40:0/2493849018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5840a5fb0 0x7fd58414f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:49.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.641+0000 7fd582ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:51:49.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.641+0000 7fd582ffd700 1 monclient: found mon.noname-b 2023-09-30T00:51:49.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.641+0000 7fd582ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:49.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.641+0000 7fd582ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:51:49.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.641+0000 7fd582ffd700 1 -- 172.21.15.40:0/2493849018 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c0094f0 con 0x7fd5840af410 2023-09-30T00:51:49.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.644+0000 7fd582ffd700 1 --2- 172.21.15.40:0/2493849018 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5840af410 0x7fd5841510b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd5840a6500 tx=0x7fd56c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:49.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.644+0000 7fd57b7fe700 10 client.?.objecter ms_handle_connect 0x7fd5840af410 2023-09-30T00:51:49.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.644+0000 7fd57b7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:49.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd57b7fe700 1 -- 172.21.15.40:0/2493849018 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c000d10 con 0x7fd5840af410 2023-09-30T00:51:49.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd57b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:49.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd57b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:49.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd57b7fe700 10 monclient: dump: 2023-09-30T00:51:49.649 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:49.650 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:49.650 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:49.650 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:49.650 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:49.650 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:49.650 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:49.650 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:49.650 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:49.650 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:49.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd57b7fe700 10 monclient: _finish_auth 0 2023-09-30T00:51:49.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd57b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:49.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd57b7fe700 1 -- 172.21.15.40:0/2493849018 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56c00bbe0 con 0x7fd5840af410 2023-09-30T00:51:49.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd589fe2700 5 monclient: authenticate success, global_id 4741 2023-09-30T00:51:49.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd589fe2700 10 monclient: _renew_subs 2023-09-30T00:51:49.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd589fe2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:49.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd589fe2700 1 -- 172.21.15.40:0/2493849018 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58414fbf0 con 0x7fd5840af410 2023-09-30T00:51:49.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd57b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:49.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd57b7fe700 1 -- 172.21.15.40:0/2493849018 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c011670 con 0x7fd5840af410 2023-09-30T00:51:49.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd57b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:49.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd57b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:49.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd57b7fe700 10 monclient: dump: 2023-09-30T00:51:49.651 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:49.651 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:49.651 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:49.652 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:49.652 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:49.652 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:49.652 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:49.652 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:49.652 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:49.652 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:49.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd581ffb700 4 set_mon_vals no callback set 2023-09-30T00:51:49.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd589fe2700 10 client.4741.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:49.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd589fe2700 10 monclient: _renew_subs 2023-09-30T00:51:49.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd589fe2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:49.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd589fe2700 1 -- 172.21.15.40:0/2493849018 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd584151f90 con 0x7fd5840af410 2023-09-30T00:51:49.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.648+0000 7fd589fe2700 1 librados: init done 2023-09-30T00:51:49.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.649+0000 7fd589fe2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:49.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.649+0000 7fd589fe2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:49.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.649+0000 7fd589fe2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:49.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.649+0000 7fd589fe2700 1 -- 172.21.15.40:0/2493849018 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd584064a00 con 0x7fd5840af410 2023-09-30T00:51:49.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.667+0000 7fd57b7fe700 1 -- 172.21.15.40:0/2493849018 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd56c011aa0 con 0x7fd5840af410 2023-09-30T00:51:49.668 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.668+0000 7fd57b7fe700 1 --2- 172.21.15.40:0/2493849018 >> v2:172.21.15.135:6800/116426 conn(0x7fd564068960 0x7fd56406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:49.668 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.668+0000 7fd5837fe700 1 --2- 172.21.15.40:0/2493849018 >> v2:172.21.15.135:6800/116426 conn(0x7fd564068960 0x7fd56406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:49.668 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.668+0000 7fd5837fe700 10 monclient: get_auth_request con 0x7fd564068960 auth_method 0 2023-09-30T00:51:49.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.668+0000 7fd57b7fe700 10 client.4741.objecter ms_handle_connect 0x7fd564068960 2023-09-30T00:51:49.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.668+0000 7fd5837fe700 1 --2- 172.21.15.40:0/2493849018 >> v2:172.21.15.135:6800/116426 conn(0x7fd564068960 0x7fd56406ae10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009d10 tx=0x7fd57c0091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.843+0000 7fd57b7fe700 1 -- 172.21.15.40:0/2493849018 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd56c057500 con 0x7fd5840af410 2023-09-30T00:51:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.843+0000 7fd57b7fe700 10 client.4741.objecter ms_dispatch 0x7fd584001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.843+0000 7fd57b7fe700 3 client.4741.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.843+0000 7fd57b7fe700 3 client.4741.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.843+0000 7fd57b7fe700 20 client.4741.objecter dump_active .. 0 homeless 2023-09-30T00:51:49.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.918+0000 7fd57b7fe700 1 -- 172.21.15.40:0/2493849018 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd56c053140 con 0x7fd5840af410 2023-09-30T00:51:49.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.918+0000 7fd57b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:49.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:49.918+0000 7fd57b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.189+0000 7fd589fe2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.189+0000 7fd589fe2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.189+0000 7fd589fe2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.189+0000 7fd589fe2700 1 -- 172.21.15.40:0/2493849018 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd5840b4250 con 0x7fd5840af410 2023-09-30T00:51:50.194 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.194+0000 7fd57b7fe700 1 -- 172.21.15.40:0/2493849018 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd56c056760 con 0x7fd5840af410 2023-09-30T00:51:50.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.194+0000 7fd57b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:50.195 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.194+0000 7fd57b7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:50.195 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:50.195 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:50.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 10 librados: watch_flush enter 2023-09-30T00:51:50.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 10 librados: watch_flush exit 2023-09-30T00:51:50.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 20 client.4741.objecter shutdown clearing up homeless session... 2023-09-30T00:51:50.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 10 client.4741.objecter successfully canceled tick 2023-09-30T00:51:50.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 1 -- 172.21.15.40:0/2493849018 >> v2:172.21.15.135:6800/116426 conn(0x7fd564068960 msgr2=0x7fd56406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:50.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 1 --2- 172.21.15.40:0/2493849018 >> v2:172.21.15.135:6800/116426 conn(0x7fd564068960 0x7fd56406ae10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009d10 tx=0x7fd57c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:51:50.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 10 monclient: shutdown 2023-09-30T00:51:50.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:50.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 1 -- 172.21.15.40:0/2493849018 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5840af410 msgr2=0x7fd5841510b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:50.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 1 --2- 172.21.15.40:0/2493849018 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5840af410 0x7fd5841510b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd5840a6500 tx=0x7fd56c017000 comp rx=0 tx=0).stop 2023-09-30T00:51:50.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 1 -- 172.21.15.40:0/2493849018 shutdown_connections 2023-09-30T00:51:50.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 1 --2- 172.21.15.40:0/2493849018 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd5840b1460 0x7fd58414ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:50.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 1 --2- 172.21.15.40:0/2493849018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5840a5fb0 0x7fd58414f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:50.198 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 1 --2- 172.21.15.40:0/2493849018 >> v2:172.21.15.135:6800/116426 conn(0x7fd564068960 0x7fd56406ae10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:50.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 1 --2- 172.21.15.40:0/2493849018 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd5840af410 0x7fd5841510b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:50.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 1 -- 172.21.15.40:0/2493849018 >> 172.21.15.40:0/2493849018 conn(0x7fd5840631f0 msgr2=0x7fd584139d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:50.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 1 -- 172.21.15.40:0/2493849018 shutdown_connections 2023-09-30T00:51:50.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 1 -- 172.21.15.40:0/2493849018 wait complete. 2023-09-30T00:51:50.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:50.196+0000 7fd589fe2700 1 librados: shutdown 2023-09-30T00:51:53.212 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:53.348 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.348+0000 7ff25c58f700 10 monclient: get_monmap_and_config 2023-09-30T00:51:53.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.348+0000 7ff25c58f700 10 monclient: build_initial_monmap 2023-09-30T00:51:53.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.348+0000 7ff25c58f700 10 monclient: monmap: 2023-09-30T00:51:53.349 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:53.349 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:53.349 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:53.349316+0000 2023-09-30T00:51:53.349 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:53.349316+0000 2023-09-30T00:51:53.349 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:53.349 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:53.350 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:53.350 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:53.350 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:53.350 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:53.350 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:53.350 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:53.352 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.348+0000 7ff25c58f700 1 Processor -- start 2023-09-30T00:51:53.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.349+0000 7ff25c58f700 1 -- start start 2023-09-30T00:51:53.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.349+0000 7ff25c58f700 10 monclient: init 2023-09-30T00:51:53.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.349+0000 7ff25c58f700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:53.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.349+0000 7ff25c58f700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-09-30T00:51:53.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.349+0000 7ff25c58f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2540b1460 0x7ff2540b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:53.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.349+0000 7ff25c58f700 10 monclient(hunting): picked mon.noname-b con 0x7ff2540b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:53.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.349+0000 7ff25c58f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff2540a5fb0 addr v1:172.21.15.40:6789/0 2023-09-30T00:51:53.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.349+0000 7ff25c58f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff25413ac20 addr v1:172.21.15.135:6789/0 2023-09-30T00:51:53.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.349+0000 7ff25c58f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2540a5b20 con 0x7ff2540a5fb0 2023-09-30T00:51:53.353 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.349+0000 7ff25c58f700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2540a71f0 con 0x7ff25413ac20 2023-09-30T00:51:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.349+0000 7ff25c58f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.349+0000 7ff25c58f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2540a74d0 con 0x7ff2540b1460 2023-09-30T00:51:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.349+0000 7ff25c58f700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.349+0000 7ff25c58f700 10 monclient(hunting): authenticate will time out at 5574.650984s 2023-09-30T00:51:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.350+0000 7ff25a32b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2540b1460 0x7ff2540b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.350+0000 7ff259b2a700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff2540a5fb0 0x7ff254141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38508/0 (socket says 172.21.15.40:38508) 2023-09-30T00:51:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.350+0000 7ff259b2a700 1 -- 172.21.15.40:0/2475489965 learned_addr learned my addr 172.21.15.40:0/2475489965 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:51:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.350+0000 7ff25a32b700 10 monclient(hunting): get_auth_request con 0x7ff2540b1460 auth_method 0 2023-09-30T00:51:53.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.350+0000 7ff25a32b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:53.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.350+0000 7ff25a32b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:53.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.350+0000 7ff25a32b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:53.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.351+0000 7ff25a32b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:53.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.351+0000 7ff25a32b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:53.355 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.351+0000 7ff25a32b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff259329700 1 -- 172.21.15.40:0/2475489965 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318817959 0 0) 0x7ff2540a5b20 con 0x7ff2540a5fb0 2023-09-30T00:51:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff259329700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff259329700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:53.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff259329700 10 monclient(hunting): my global_id is 4674 2023-09-30T00:51:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff259329700 1 -- 172.21.15.40:0/2475489965 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff240003570 con 0x7ff2540a5fb0 2023-09-30T00:51:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff259329700 1 -- 172.21.15.40:0/2475489965 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452443362 0 0) 0x7ff2540a71f0 con 0x7ff25413ac20 2023-09-30T00:51:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff259329700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff259329700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff259329700 10 monclient(hunting): my global_id is 4744 2023-09-30T00:51:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff259329700 1 -- 172.21.15.40:0/2475489965 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2540a5b20 con 0x7ff25413ac20 2023-09-30T00:51:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff25a32b700 10 monclient(hunting): handle_auth_done global_id 4598 payload 274 2023-09-30T00:51:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff25a32b700 1 -- 172.21.15.40:0/2475489965 >> v1:172.21.15.135:6789/0 conn(0x7ff25413ac20 legacy=0x7ff25413b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff25a32b700 1 -- 172.21.15.40:0/2475489965 >> v1:172.21.15.40:6789/0 conn(0x7ff2540a5fb0 legacy=0x7ff254141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff25a32b700 10 monclient: _finish_hunting 0 2023-09-30T00:51:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff25a32b700 1 monclient: found mon.noname-b 2023-09-30T00:51:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff25a32b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff25a32b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:51:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.354+0000 7ff25a32b700 1 -- 172.21.15.40:0/2475489965 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2540a63c0 con 0x7ff2540b1460 2023-09-30T00:51:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.356+0000 7ff25a32b700 1 --2- 172.21.15.40:0/2475489965 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2540b1460 0x7ff2540b1830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff250009910 tx=0x7ff2500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0fa62d688f7bc5a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:53.365 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff259329700 1 -- 172.21.15.40:0/2475489965 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff250011630 con 0x7ff2540b1460 2023-09-30T00:51:53.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff259329700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:53.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff259329700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:53.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff259329700 10 monclient: dump: 2023-09-30T00:51:53.366 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:53.366 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:53.366 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:53.366 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:53.366 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:53.366 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:53.367 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff259329700 10 monclient: _finish_auth 0 2023-09-30T00:51:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff259329700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff259329700 1 -- 172.21.15.40:0/2475489965 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff250011790 con 0x7ff2540b1460 2023-09-30T00:51:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff259329700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff259329700 1 -- 172.21.15.40:0/2475489965 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff250011b30 con 0x7ff2540b1460 2023-09-30T00:51:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff259329700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff259329700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:51:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff259329700 10 monclient: dump: 2023-09-30T00:51:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:53.368 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff25c58f700 5 monclient: authenticate success, global_id 4598 2023-09-30T00:51:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff25c58f700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.360+0000 7ff25c58f700 4 set_mon_vals no callback set 2023-09-30T00:51:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.361+0000 7ff25c58f700 10 monclient: shutdown 2023-09-30T00:51:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.361+0000 7ff25c58f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.361+0000 7ff25c58f700 1 -- 172.21.15.40:0/2475489965 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2540b1460 msgr2=0x7ff2540b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.361+0000 7ff25c58f700 1 --2- 172.21.15.40:0/2475489965 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2540b1460 0x7ff2540b1830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff250009910 tx=0x7ff2500091a0 comp rx=0 tx=0).stop 2023-09-30T00:51:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.361+0000 7ff25c58f700 1 -- 172.21.15.40:0/2475489965 shutdown_connections 2023-09-30T00:51:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.361+0000 7ff25c58f700 1 --2- 172.21.15.40:0/2475489965 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff2540b1460 0x7ff2540b1830 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.361+0000 7ff25c58f700 1 -- 172.21.15.40:0/2475489965 >> 172.21.15.40:0/2475489965 conn(0x7ff2540631f0 msgr2=0x7ff2540635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.361+0000 7ff25c58f700 1 -- 172.21.15.40:0/2475489965 shutdown_connections 2023-09-30T00:51:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.361+0000 7ff25c58f700 1 -- 172.21.15.40:0/2475489965 wait complete. 2023-09-30T00:51:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.361+0000 7ff25c58f700 10 monclient: build_initial_monmap 2023-09-30T00:51:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.361+0000 7ff25c58f700 10 monclient: monmap: 2023-09-30T00:51:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:53.371 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.362+0000 7ff25c58f700 1 librados: starting msgr at 2023-09-30T00:51:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.362+0000 7ff25c58f700 1 librados: starting objecter 2023-09-30T00:51:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.362+0000 7ff25c58f700 1 Processor -- start 2023-09-30T00:51:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.362+0000 7ff25c58f700 1 -- start start 2023-09-30T00:51:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.362+0000 7ff25c58f700 1 librados: setting wanted keys 2023-09-30T00:51:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.362+0000 7ff25c58f700 1 librados: calling monclient init 2023-09-30T00:51:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.362+0000 7ff25c58f700 10 monclient: init 2023-09-30T00:51:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:51:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2540a5fb0 0x7ff2540a9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 10 monclient(hunting): picked mon.noname-a con 0x7ff2540a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2540b1460 0x7ff2540a8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:53.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 10 monclient(hunting): picked mon.noname-b con 0x7ff2540b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:53.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff25413ac20 0x7ff2540ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:53.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 10 monclient(hunting): picked mon.noname-c con 0x7ff25413ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:53.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:53.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25414d750 con 0x7ff2540a5fb0 2023-09-30T00:51:53.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:53.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2540aa970 con 0x7ff2540b1460 2023-09-30T00:51:53.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:53.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2540a79f0 con 0x7ff25413ac20 2023-09-30T00:51:53.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:53.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25c58f700 10 monclient(hunting): authenticate will time out at 5574.664318s 2023-09-30T00:51:53.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.363+0000 7ff25ab2c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff25413ac20 0x7ff2540ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:53.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff259b2a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2540b1460 0x7ff2540a8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:53.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff25a32b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2540a5fb0 0x7ff2540a9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:53.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff25ab2c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff25413ac20 0x7ff2540ae500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:42808/0 (socket says 172.21.15.40:42808) 2023-09-30T00:51:53.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff25ab2c700 1 -- 172.21.15.40:0/130490973 learned_addr learned my addr 172.21.15.40:0/130490973 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:53.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff25ab2c700 10 monclient(hunting): get_auth_request con 0x7ff25413ac20 auth_method 0 2023-09-30T00:51:53.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff25ab2c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:53.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff25ab2c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:53.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff25ab2c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:53.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff25a32b700 10 monclient(hunting): get_auth_request con 0x7ff2540a5fb0 auth_method 0 2023-09-30T00:51:53.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff25a32b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:53.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff25a32b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:53.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff25a32b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:53.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff25ab2c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:53.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff25ab2c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:53.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff25ab2c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff259b2a700 10 monclient(hunting): get_auth_request con 0x7ff2540b1460 auth_method 0 2023-09-30T00:51:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff259b2a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff259b2a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.364+0000 7ff259b2a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.365+0000 7ff25a32b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.365+0000 7ff25a32b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.365+0000 7ff25a32b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.365+0000 7ff259b2a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.365+0000 7ff259b2a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.365+0000 7ff259b2a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:53.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.368+0000 7ff25ab2c700 10 monclient(hunting): handle_auth_done global_id 4601 payload 824 2023-09-30T00:51:53.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.368+0000 7ff25ab2c700 1 -- 172.21.15.40:0/130490973 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2540b1460 msgr2=0x7ff2540a8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:53.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.368+0000 7ff25ab2c700 1 --2- 172.21.15.40:0/130490973 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2540b1460 0x7ff2540a8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:53.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.368+0000 7ff25ab2c700 1 -- 172.21.15.40:0/130490973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2540a5fb0 msgr2=0x7ff2540a9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:53.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.368+0000 7ff25ab2c700 1 --2- 172.21.15.40:0/130490973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2540a5fb0 0x7ff2540a9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:53.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.368+0000 7ff25ab2c700 10 monclient: _finish_hunting 0 2023-09-30T00:51:53.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.368+0000 7ff25ab2c700 1 monclient: found mon.noname-c 2023-09-30T00:51:53.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.368+0000 7ff25ab2c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:53.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.368+0000 7ff25ab2c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:51:53.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.368+0000 7ff25ab2c700 1 -- 172.21.15.40:0/130490973 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff250009420 con 0x7ff25413ac20 2023-09-30T00:51:53.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.372+0000 7ff25ab2c700 1 --2- 172.21.15.40:0/130490973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff25413ac20 0x7ff2540ae500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00c890 tx=0x7ff24c002700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:53.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.372+0000 7ff2467fc700 10 client.?.objecter ms_handle_connect 0x7ff25413ac20 2023-09-30T00:51:53.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.372+0000 7ff2467fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:53.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff2467fc700 1 -- 172.21.15.40:0/130490973 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c012630 con 0x7ff25413ac20 2023-09-30T00:51:53.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff2467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:53.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff2467fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:51:53.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff2467fc700 10 monclient: dump: 2023-09-30T00:51:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:53.381 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff2467fc700 10 monclient: _finish_auth 0 2023-09-30T00:51:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff2467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff25c58f700 5 monclient: authenticate success, global_id 4601 2023-09-30T00:51:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff25c58f700 10 monclient: _renew_subs 2023-09-30T00:51:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff25c58f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff25c58f700 1 -- 172.21.15.40:0/130490973 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2540aabe0 con 0x7ff25413ac20 2023-09-30T00:51:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff25c58f700 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff25c58f700 10 monclient: _renew_subs 2023-09-30T00:51:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff25c58f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff25c58f700 1 -- 172.21.15.40:0/130490973 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2540aea40 con 0x7ff25413ac20 2023-09-30T00:51:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.375+0000 7ff25c58f700 1 librados: init done 2023-09-30T00:51:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.376+0000 7ff25c58f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.376+0000 7ff25c58f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.376+0000 7ff25c58f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.376+0000 7ff25c58f700 1 -- 172.21.15.40:0/130490973 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25414d8c0 con 0x7ff25413ac20 2023-09-30T00:51:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.377+0000 7ff2467fc700 1 -- 172.21.15.40:0/130490973 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff24c0129a0 con 0x7ff25413ac20 2023-09-30T00:51:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.377+0000 7ff2467fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.377+0000 7ff2467fc700 1 -- 172.21.15.40:0/130490973 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24c012b10 con 0x7ff25413ac20 2023-09-30T00:51:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.377+0000 7ff2467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.377+0000 7ff2467fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:51:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.377+0000 7ff2467fc700 10 monclient: dump: 2023-09-30T00:51:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:53.386 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.377+0000 7ff258b28700 4 set_mon_vals no callback set 2023-09-30T00:51:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.642+0000 7ff2467fc700 1 -- 172.21.15.40:0/130490973 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ff24c012c70 con 0x7ff25413ac20 2023-09-30T00:51:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.643+0000 7ff2467fc700 1 --2- 172.21.15.40:0/130490973 >> v2:172.21.15.135:6800/116426 conn(0x7ff238068890 0x7ff23806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:53.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.643+0000 7ff25a32b700 1 --2- 172.21.15.40:0/130490973 >> v2:172.21.15.135:6800/116426 conn(0x7ff238068890 0x7ff23806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:53.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.643+0000 7ff25a32b700 10 monclient: get_auth_request con 0x7ff238068890 auth_method 0 2023-09-30T00:51:53.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.644+0000 7ff25a32b700 1 --2- 172.21.15.40:0/130490973 >> v2:172.21.15.135:6800/116426 conn(0x7ff238068890 0x7ff23806ad40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff2500093f0 tx=0x7ff250017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:53.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.644+0000 7ff2467fc700 10 client.4601.objecter ms_handle_connect 0x7ff238068890 2023-09-30T00:51:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.670+0000 7ff2467fc700 1 -- 172.21.15.40:0/130490973 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff24c0102e0 con 0x7ff25413ac20 2023-09-30T00:51:53.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.671+0000 7ff2467fc700 10 client.4601.objecter ms_dispatch 0x7ff254001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.671+0000 7ff2467fc700 3 client.4601.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.671+0000 7ff2467fc700 3 client.4601.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.671+0000 7ff2467fc700 20 client.4601.objecter dump_active .. 0 homeless 2023-09-30T00:51:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.671+0000 7ff2467fc700 1 -- 172.21.15.40:0/130490973 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff24c051d20 con 0x7ff25413ac20 2023-09-30T00:51:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.671+0000 7ff2467fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:53.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.671+0000 7ff2467fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:53.735 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:05:29.446 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:53.800 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:05:29.511 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:53.802 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:29.489 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:53.807 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:29.493 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:53.817 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:05:29.529 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:53.819 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:05:29.531 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:53.834 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:29.521 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:53.856 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:05:29.567 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:53.858 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:05:29.570 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:51:53.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.954+0000 7ff25c58f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:53.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.954+0000 7ff25c58f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:53.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.954+0000 7ff25c58f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:51:53.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.954+0000 7ff25c58f700 1 -- 172.21.15.40:0/130490973 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff254064a00 con 0x7ff25413ac20 2023-09-30T00:51:53.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.975+0000 7ff2467fc700 1 -- 172.21.15.40:0/130490973 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff24c055340 con 0x7ff25413ac20 2023-09-30T00:51:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.975+0000 7ff2467fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:53.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.975+0000 7ff2467fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:53.976 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:53.976 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.977+0000 7ff25c58f700 10 librados: watch_flush enter 2023-09-30T00:51:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.977+0000 7ff25c58f700 10 librados: watch_flush exit 2023-09-30T00:51:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.977+0000 7ff25c58f700 20 client.4601.objecter shutdown clearing up homeless session... 2023-09-30T00:51:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.977+0000 7ff25c58f700 10 client.4601.objecter successfully canceled tick 2023-09-30T00:51:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 1 -- 172.21.15.40:0/130490973 >> v2:172.21.15.135:6800/116426 conn(0x7ff238068890 msgr2=0x7ff23806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 1 --2- 172.21.15.40:0/130490973 >> v2:172.21.15.135:6800/116426 conn(0x7ff238068890 0x7ff23806ad40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff2500093f0 tx=0x7ff250017000 comp rx=0 tx=0).stop 2023-09-30T00:51:53.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 10 monclient: shutdown 2023-09-30T00:51:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 1 -- 172.21.15.40:0/130490973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff25413ac20 msgr2=0x7ff2540ae500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 1 --2- 172.21.15.40:0/130490973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff25413ac20 0x7ff2540ae500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00c890 tx=0x7ff24c002700 comp rx=0 tx=0).stop 2023-09-30T00:51:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 1 -- 172.21.15.40:0/130490973 shutdown_connections 2023-09-30T00:51:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 1 --2- 172.21.15.40:0/130490973 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7ff25413ac20 0x7ff2540ae500 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 1 --2- 172.21.15.40:0/130490973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2540a5fb0 0x7ff2540a9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 1 --2- 172.21.15.40:0/130490973 >> v2:172.21.15.135:6800/116426 conn(0x7ff238068890 0x7ff23806ad40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 1 --2- 172.21.15.40:0/130490973 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7ff2540b1460 0x7ff2540a8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 1 -- 172.21.15.40:0/130490973 >> 172.21.15.40:0/130490973 conn(0x7ff2540631f0 msgr2=0x7ff2540b3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 1 -- 172.21.15.40:0/130490973 shutdown_connections 2023-09-30T00:51:53.980 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 1 -- 172.21.15.40:0/130490973 wait complete. 2023-09-30T00:51:53.981 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:53.978+0000 7ff25c58f700 1 librados: shutdown 2023-09-30T00:51:56.992 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:51:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.134+0000 7f11fa846700 10 monclient: get_monmap_and_config 2023-09-30T00:51:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.134+0000 7f11fa846700 10 monclient: build_initial_monmap 2023-09-30T00:51:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.134+0000 7f11fa846700 10 monclient: monmap: 2023-09-30T00:51:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:51:57.135678+0000 2023-09-30T00:51:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:51:57.135678+0000 2023-09-30T00:51:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:51:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:51:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:51:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:51:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:51:57.136 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.135+0000 7f11fa846700 1 Processor -- start 2023-09-30T00:51:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.135+0000 7f11fa846700 1 -- start start 2023-09-30T00:51:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.135+0000 7f11fa846700 10 monclient: init 2023-09-30T00:51:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.136+0000 7f11fa846700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.136+0000 7f11fa846700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-09-30T00:51:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.136+0000 7f11fa846700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f11f40a5710 0x7f11f40a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.136+0000 7f11fa846700 10 monclient(hunting): picked mon.noname-a con 0x7f11f40a5710 addr v2:172.21.15.40:3300/0 2023-09-30T00:51:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.136+0000 7f11fa846700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f11f40a7b20 addr v1:172.21.15.40:6789/0 2023-09-30T00:51:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.136+0000 7f11fa846700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f11f413e8e0 0x7f11f413ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.136+0000 7f11fa846700 10 monclient(hunting): picked mon.noname-c con 0x7f11f413e8e0 addr v2:172.21.15.135:3300/0 2023-09-30T00:51:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.136+0000 7f11fa846700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11f40a7ff0 con 0x7f11f40a7b20 2023-09-30T00:51:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.136+0000 7f11fa846700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.136+0000 7f11fa846700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f40aedc0 con 0x7f11f40a5710 2023-09-30T00:51:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.136+0000 7f11fa846700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.136+0000 7f11fa846700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11f40aef00 con 0x7f11f413e8e0 2023-09-30T00:51:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.136+0000 7f11fa846700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.136+0000 7f11fa846700 10 monclient(hunting): authenticate will time out at 5578.437327s 2023-09-30T00:51:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.144+0000 7f11f8de3700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f11f413e8e0 0x7f11f413ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.145+0000 7f11f8de3700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f11f413e8e0 0x7f11f413ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:34784/0 (socket says 172.21.15.40:34784) 2023-09-30T00:51:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.145+0000 7f11f8de3700 1 -- 172.21.15.40:0/147442550 learned_addr learned my addr 172.21.15.40:0/147442550 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.145+0000 7f11f8de3700 10 monclient(hunting): get_auth_request con 0x7f11f413e8e0 auth_method 0 2023-09-30T00:51:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.145+0000 7f11f8de3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.145+0000 7f11f8de3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.145+0000 7f11f8de3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.145+0000 7f11f8de3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.145+0000 7f11f8de3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.145+0000 7f11f8de3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.147+0000 7f11f3fff700 1 --2- 172.21.15.40:0/147442550 >> v2:172.21.15.40:3300/0 conn(0x7f11f40a5710 0x7f11f40a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.148+0000 7f11f3fff700 10 monclient(hunting): get_auth_request con 0x7f11f40a5710 auth_method 0 2023-09-30T00:51:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.148+0000 7f11f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.148+0000 7f11f3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.148+0000 7f11f3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.148+0000 7f11f8de3700 10 monclient(hunting): handle_auth_done global_id 4750 payload 274 2023-09-30T00:51:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.148+0000 7f11f8de3700 1 -- 172.21.15.40:0/147442550 >> v2:172.21.15.40:3300/0 conn(0x7f11f40a5710 msgr2=0x7f11f40a75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:51:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.149+0000 7f11f8de3700 1 --2- 172.21.15.40:0/147442550 >> v2:172.21.15.40:3300/0 conn(0x7f11f40a5710 0x7f11f40a75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.149+0000 7f11f8de3700 1 -- 172.21.15.40:0/147442550 >> v1:172.21.15.40:6789/0 conn(0x7f11f40a7b20 legacy=0x7f11f413e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.149+0000 7f11f8de3700 10 monclient: _finish_hunting 0 2023-09-30T00:51:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.149+0000 7f11f8de3700 1 monclient: found mon.noname-c 2023-09-30T00:51:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.149+0000 7f11f8de3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.149+0000 7f11f8de3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:51:57.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.149+0000 7f11f8de3700 1 -- 172.21.15.40:0/147442550 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f413ed80 con 0x7f11f413e8e0 2023-09-30T00:51:57.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.151+0000 7f11f8de3700 1 --2- 172.21.15.40:0/147442550 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11f413e8e0 0x7f11f413ae40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f11f413bf20 tx=0x7f11ec009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75c3e92c4af4ed68 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:57.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.154+0000 7f11f2ffd700 1 -- 172.21.15.40:0/147442550 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec00e070 con 0x7f11f413e8e0 2023-09-30T00:51:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.154+0000 7f11f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.154+0000 7f11f2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:51:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.154+0000 7f11f2ffd700 10 monclient: dump: 2023-09-30T00:51:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:57.156 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.154+0000 7f11f2ffd700 10 monclient: _finish_auth 0 2023-09-30T00:51:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.154+0000 7f11f2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.154+0000 7f11fa846700 5 monclient: authenticate success, global_id 4750 2023-09-30T00:51:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.154+0000 7f11fa846700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:51:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11f2ffd700 1 -- 172.21.15.40:0/147442550 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ec00ba50 con 0x7f11f413e8e0 2023-09-30T00:51:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11f2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11f2ffd700 1 -- 172.21.15.40:0/147442550 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec0092e0 con 0x7f11f413e8e0 2023-09-30T00:51:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11fa846700 10 monclient: get_monmap_and_config success 2023-09-30T00:51:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11fa846700 4 set_mon_vals no callback set 2023-09-30T00:51:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11f2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11f2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11f2ffd700 10 monclient: dump: 2023-09-30T00:51:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:57.159 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11fa846700 10 monclient: shutdown 2023-09-30T00:51:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11fa846700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11fa846700 1 -- 172.21.15.40:0/147442550 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11f413e8e0 msgr2=0x7f11f413ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11fa846700 1 --2- 172.21.15.40:0/147442550 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11f413e8e0 0x7f11f413ae40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f11f413bf20 tx=0x7f11ec009840 comp rx=0 tx=0).stop 2023-09-30T00:51:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11fa846700 1 -- 172.21.15.40:0/147442550 shutdown_connections 2023-09-30T00:51:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11fa846700 1 --2- 172.21.15.40:0/147442550 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11f413e8e0 0x7f11f413ae40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11fa846700 1 --2- 172.21.15.40:0/147442550 >> v2:172.21.15.40:3300/0 conn(0x7f11f40a5710 0x7f11f40a75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11fa846700 1 -- 172.21.15.40:0/147442550 >> 172.21.15.40:0/147442550 conn(0x7f11f4069080 msgr2=0x7f11f40631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11fa846700 1 -- 172.21.15.40:0/147442550 shutdown_connections 2023-09-30T00:51:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.158+0000 7f11fa846700 1 -- 172.21.15.40:0/147442550 wait complete. 2023-09-30T00:51:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.159+0000 7f11fa846700 10 monclient: build_initial_monmap 2023-09-30T00:51:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.159+0000 7f11fa846700 10 monclient: monmap: 2023-09-30T00:51:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:51:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:51:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:51:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:51:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:51:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:51:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:51:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:51:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:51:57.164 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.159+0000 7f11fa846700 1 librados: starting msgr at 2023-09-30T00:51:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.159+0000 7f11fa846700 1 librados: starting objecter 2023-09-30T00:51:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 1 Processor -- start 2023-09-30T00:51:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 1 -- start start 2023-09-30T00:51:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 1 librados: setting wanted keys 2023-09-30T00:51:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 1 librados: calling monclient init 2023-09-30T00:51:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 10 monclient: init 2023-09-30T00:51:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 10 monclient: _reopen_session rank -1 2023-09-30T00:51:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:51:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11f40a5710 0x7f11f40aced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:57.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 10 monclient(hunting): picked mon.noname-a con 0x7f11f40a5710 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:51:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11f40a7b20 0x7f11f40adc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 10 monclient(hunting): picked mon.noname-c con 0x7f11f40a7b20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:51:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11f413e8e0 0x7f11f40aa560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 10 monclient(hunting): picked mon.noname-b con 0x7f11f413e8e0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:51:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f40b2b90 con 0x7f11f40a5710 2023-09-30T00:51:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f40ae150 con 0x7f11f413e8e0 2023-09-30T00:51:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 10 monclient(hunting): start opening mon connection 2023-09-30T00:51:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11f40aa040 con 0x7f11f40a7b20 2023-09-30T00:51:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 10 monclient(hunting): _renew_subs 2023-09-30T00:51:57.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.160+0000 7f11fa846700 10 monclient(hunting): authenticate will time out at 5578.461882s 2023-09-30T00:51:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f3fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11f40a5710 0x7f11f40aced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f37fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11f40a7b20 0x7f11f40adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f8de3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11f413e8e0 0x7f11f40aa560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f3fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11f40a5710 0x7f11f40aced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60942/0 (socket says 172.21.15.40:60942) 2023-09-30T00:51:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f3fff700 1 -- 172.21.15.40:0/3645481058 learned_addr learned my addr 172.21.15.40:0/3645481058 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:51:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f8de3700 10 monclient(hunting): get_auth_request con 0x7f11f413e8e0 auth_method 0 2023-09-30T00:51:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f8de3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f8de3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f8de3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f3fff700 10 monclient(hunting): get_auth_request con 0x7f11f40a5710 auth_method 0 2023-09-30T00:51:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:57.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f37fe700 10 monclient(hunting): get_auth_request con 0x7f11f40a7b20 auth_method 0 2023-09-30T00:51:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:51:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:51:57.167 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.161+0000 7f11f37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:51:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.162+0000 7f11f8de3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.162+0000 7f11f8de3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.162+0000 7f11f8de3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.162+0000 7f11f3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.162+0000 7f11f3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.162+0000 7f11f3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.162+0000 7f11f37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:51:57.168 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.162+0000 7f11f37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:51:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.162+0000 7f11f37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:51:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.165+0000 7f11f8de3700 10 monclient(hunting): handle_auth_done global_id 4753 payload 824 2023-09-30T00:51:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.166+0000 7f11f8de3700 1 -- 172.21.15.40:0/3645481058 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11f40a7b20 msgr2=0x7f11f40adc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.166+0000 7f11f8de3700 1 --2- 172.21.15.40:0/3645481058 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11f40a7b20 0x7f11f40adc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.166+0000 7f11f8de3700 1 -- 172.21.15.40:0/3645481058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11f40a5710 msgr2=0x7f11f40aced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.166+0000 7f11f8de3700 1 --2- 172.21.15.40:0/3645481058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11f40a5710 0x7f11f40aced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.166+0000 7f11f8de3700 10 monclient: _finish_hunting 0 2023-09-30T00:51:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.166+0000 7f11f8de3700 1 monclient: found mon.noname-b 2023-09-30T00:51:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.166+0000 7f11f8de3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:51:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.166+0000 7f11f8de3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:51:57.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.166+0000 7f11f8de3700 1 -- 172.21.15.40:0/3645481058 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ec009ae0 con 0x7f11f413e8e0 2023-09-30T00:51:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.168+0000 7f11f8de3700 1 --2- 172.21.15.40:0/3645481058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11f413e8e0 0x7f11f40aa560 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f11ec015000 tx=0x7f11ec018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.168+0000 7f11dbfff700 10 client.?.objecter ms_handle_connect 0x7f11f413e8e0 2023-09-30T00:51:57.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.168+0000 7f11dbfff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:51:57.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11dbfff700 1 -- 172.21.15.40:0/3645481058 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec00be60 con 0x7f11f413e8e0 2023-09-30T00:51:57.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:57.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11dbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:51:57.173 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11dbfff700 10 monclient: dump: 2023-09-30T00:51:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:57.174 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11dbfff700 10 monclient: _finish_auth 0 2023-09-30T00:51:57.174 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11dbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:51:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11dbfff700 1 -- 172.21.15.40:0/3645481058 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ec008d20 con 0x7f11f413e8e0 2023-09-30T00:51:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11dbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:51:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11dbfff700 1 -- 172.21.15.40:0/3645481058 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec0256b0 con 0x7f11f413e8e0 2023-09-30T00:51:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11fa846700 5 monclient: authenticate success, global_id 4753 2023-09-30T00:51:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11fa846700 10 monclient: _renew_subs 2023-09-30T00:51:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11fa846700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11fa846700 1 -- 172.21.15.40:0/3645481058 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f40a9cd0 con 0x7f11f413e8e0 2023-09-30T00:51:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:51:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11dbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:51:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11dbfff700 10 monclient: dump: 2023-09-30T00:51:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:51:57.175 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:51:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:51:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:51:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:51:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:51:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:51:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:51:57.176 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:51:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11f27fc700 4 set_mon_vals no callback set 2023-09-30T00:51:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11fa846700 10 client.4753.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:51:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11fa846700 10 monclient: _renew_subs 2023-09-30T00:51:57.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11fa846700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11fa846700 1 -- 172.21.15.40:0/3645481058 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f40aad90 con 0x7f11f413e8e0 2023-09-30T00:51:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.172+0000 7f11fa846700 1 librados: init done 2023-09-30T00:51:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.173+0000 7f11fa846700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.173+0000 7f11fa846700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.173+0000 7f11fa846700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.173+0000 7f11fa846700 1 -- 172.21.15.40:0/3645481058 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f4072b90 con 0x7f11f413e8e0 2023-09-30T00:51:57.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.422+0000 7f11dbfff700 1 -- 172.21.15.40:0/3645481058 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f11ec003040 con 0x7f11f413e8e0 2023-09-30T00:51:57.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.422+0000 7f11dbfff700 1 --2- 172.21.15.40:0/3645481058 >> v2:172.21.15.135:6800/116426 conn(0x7f11e0068890 0x7f11e006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:51:57.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.423+0000 7f11f3fff700 1 --2- 172.21.15.40:0/3645481058 >> v2:172.21.15.135:6800/116426 conn(0x7f11e0068890 0x7f11e006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:51:57.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.423+0000 7f11f3fff700 10 monclient: get_auth_request con 0x7f11e0068890 auth_method 0 2023-09-30T00:51:57.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.423+0000 7f11f3fff700 1 --2- 172.21.15.40:0/3645481058 >> v2:172.21.15.135:6800/116426 conn(0x7f11e0068890 0x7f11e006ad40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f11e4006fd0 tx=0x7f11e4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:51:57.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.423+0000 7f11dbfff700 10 client.4753.objecter ms_handle_connect 0x7f11e0068890 2023-09-30T00:51:57.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.427+0000 7f11dbfff700 1 -- 172.21.15.40:0/3645481058 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f11ec057250 con 0x7f11f413e8e0 2023-09-30T00:51:57.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.428+0000 7f11dbfff700 10 client.4753.objecter ms_dispatch 0x7f11f4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:51:57.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.428+0000 7f11dbfff700 3 client.4753.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:51:57.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.428+0000 7f11dbfff700 3 client.4753.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:51:57.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.428+0000 7f11dbfff700 20 client.4753.objecter dump_active .. 0 homeless 2023-09-30T00:51:57.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.440+0000 7f11dbfff700 1 -- 172.21.15.40:0/3645481058 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f11ec053590 con 0x7f11f413e8e0 2023-09-30T00:51:57.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.440+0000 7f11dbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:51:57.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.440+0000 7f11dbfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:51:57.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.694+0000 7f11fa846700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:51:57.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.694+0000 7f11fa846700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:57.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.694+0000 7f11fa846700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:51:57.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.694+0000 7f11fa846700 1 -- 172.21.15.40:0/3645481058 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f11f40aa200 con 0x7f11f413e8e0 2023-09-30T00:51:57.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.700+0000 7f11dbfff700 1 -- 172.21.15.40:0/3645481058 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f11ec020100 con 0x7f11f413e8e0 2023-09-30T00:51:57.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.700+0000 7f11dbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:51:57.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.700+0000 7f11dbfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:51:57.700 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:51:57.700 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:51:57.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.701+0000 7f11fa846700 10 librados: watch_flush enter 2023-09-30T00:51:57.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.701+0000 7f11fa846700 10 librados: watch_flush exit 2023-09-30T00:51:57.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.701+0000 7f11fa846700 20 client.4753.objecter shutdown clearing up homeless session... 2023-09-30T00:51:57.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.701+0000 7f11fa846700 10 client.4753.objecter successfully canceled tick 2023-09-30T00:51:57.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 1 -- 172.21.15.40:0/3645481058 >> v2:172.21.15.135:6800/116426 conn(0x7f11e0068890 msgr2=0x7f11e006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:57.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 1 --2- 172.21.15.40:0/3645481058 >> v2:172.21.15.135:6800/116426 conn(0x7f11e0068890 0x7f11e006ad40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f11e4006fd0 tx=0x7f11e4008040 comp rx=0 tx=0).stop 2023-09-30T00:51:57.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 10 monclient: shutdown 2023-09-30T00:51:57.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:51:57.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 1 -- 172.21.15.40:0/3645481058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11f413e8e0 msgr2=0x7f11f40aa560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:51:57.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 1 --2- 172.21.15.40:0/3645481058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11f413e8e0 0x7f11f40aa560 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f11ec015000 tx=0x7f11ec018000 comp rx=0 tx=0).stop 2023-09-30T00:51:57.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 1 -- 172.21.15.40:0/3645481058 shutdown_connections 2023-09-30T00:51:57.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 1 --2- 172.21.15.40:0/3645481058 >> v2:172.21.15.135:6800/116426 conn(0x7f11e0068890 0x7f11e006ad40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:57.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 1 --2- 172.21.15.40:0/3645481058 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f11f413e8e0 0x7f11f40aa560 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:57.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 1 --2- 172.21.15.40:0/3645481058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11f40a5710 0x7f11f40aced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:57.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 1 --2- 172.21.15.40:0/3645481058 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f11f40a7b20 0x7f11f40adc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:51:57.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 1 -- 172.21.15.40:0/3645481058 >> 172.21.15.40:0/3645481058 conn(0x7f11f4069080 msgr2=0x7f11f40afb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:51:57.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 1 -- 172.21.15.40:0/3645481058 shutdown_connections 2023-09-30T00:51:57.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 1 -- 172.21.15.40:0/3645481058 wait complete. 2023-09-30T00:51:57.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:51:57.702+0000 7f11fa846700 1 librados: shutdown 2023-09-30T00:52:00.717 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:00.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.870+0000 7f6ddefa9700 10 monclient: get_monmap_and_config 2023-09-30T00:52:00.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.870+0000 7f6ddefa9700 10 monclient: build_initial_monmap 2023-09-30T00:52:00.870 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.870+0000 7f6ddefa9700 10 monclient: monmap: 2023-09-30T00:52:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:00.871137+0000 2023-09-30T00:52:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:00.871137+0000 2023-09-30T00:52:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:00.872 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.870+0000 7f6ddefa9700 1 Processor -- start 2023-09-30T00:52:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 1 -- start start 2023-09-30T00:52:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 10 monclient: init 2023-09-30T00:52:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-09-30T00:52:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6dd80b1460 0x7f6dd80b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 10 monclient(hunting): picked mon.noname-c con 0x7f6dd80b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:52:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6dd80a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:52:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dd813ac20 0x7f6dd813b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 10 monclient(hunting): picked mon.noname-b con 0x7f6dd813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd80a5b20 con 0x7f6dd80a5fb0 2023-09-30T00:52:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dd80a71f0 con 0x7f6dd80b1460 2023-09-30T00:52:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd80a74d0 con 0x7f6dd813ac20 2023-09-30T00:52:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.871+0000 7f6ddefa9700 10 monclient(hunting): authenticate will time out at 5582.172766s 2023-09-30T00:52:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.872+0000 7f6ddd546700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dd813ac20 0x7f6dd813b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.872+0000 7f6ddcd45700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6dd80b1460 0x7f6dd80b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.872+0000 7f6ddd546700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dd813ac20 0x7f6dd813b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:42828/0 (socket says 172.21.15.40:42828) 2023-09-30T00:52:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.872+0000 7f6ddd546700 1 -- 172.21.15.40:0/2594518573 learned_addr learned my addr 172.21.15.40:0/2594518573 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.872+0000 7f6ddd546700 10 monclient(hunting): get_auth_request con 0x7f6dd813ac20 auth_method 0 2023-09-30T00:52:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.872+0000 7f6ddd546700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.872+0000 7f6ddd546700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.872+0000 7f6ddd546700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.873+0000 7f6ddcd45700 10 monclient(hunting): get_auth_request con 0x7f6dd80b1460 auth_method 0 2023-09-30T00:52:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.873+0000 7f6ddcd45700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.873+0000 7f6ddcd45700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.873+0000 7f6ddcd45700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.873+0000 7f6ddd546700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.873+0000 7f6ddd546700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.873+0000 7f6ddd546700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:00.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.874+0000 7f6ddcd45700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:00.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.874+0000 7f6ddcd45700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:00.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.874+0000 7f6ddcd45700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:00.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.875+0000 7f6ddd546700 10 monclient(hunting): handle_auth_done global_id 4607 payload 274 2023-09-30T00:52:00.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.875+0000 7f6ddd546700 1 -- 172.21.15.40:0/2594518573 >> v2:172.21.15.135:3300/0 conn(0x7f6dd80b1460 msgr2=0x7f6dd80b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:52:00.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.876+0000 7f6ddd546700 1 --2- 172.21.15.40:0/2594518573 >> v2:172.21.15.135:3300/0 conn(0x7f6dd80b1460 0x7f6dd80b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:00.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.876+0000 7f6ddd546700 1 -- 172.21.15.40:0/2594518573 >> v1:172.21.15.135:6789/0 conn(0x7f6dd80a5fb0 legacy=0x7f6dd8141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:00.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.876+0000 7f6ddd546700 10 monclient: _finish_hunting 0 2023-09-30T00:52:00.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.876+0000 7f6ddd546700 1 monclient: found mon.noname-b 2023-09-30T00:52:00.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.876+0000 7f6ddd546700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:00.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.876+0000 7f6ddd546700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:52:00.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.876+0000 7f6ddd546700 1 -- 172.21.15.40:0/2594518573 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd80a63c0 con 0x7f6dd813ac20 2023-09-30T00:52:00.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.878+0000 7f6ddd546700 1 --2- 172.21.15.40:0/2594518573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dd813ac20 0x7f6dd813b080 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009f20 tx=0x7f6dd0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6367b7b74eff6e1b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6dd77fe700 1 -- 172.21.15.40:0/2594518573 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd000e070 con 0x7f6dd813ac20 2023-09-30T00:52:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6dd77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6dd77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:52:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6dd77fe700 10 monclient: dump: 2023-09-30T00:52:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:00.886 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6dd77fe700 10 monclient: _finish_auth 0 2023-09-30T00:52:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6dd77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6dd77fe700 1 -- 172.21.15.40:0/2594518573 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd000b9d0 con 0x7f6dd813ac20 2023-09-30T00:52:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6ddefa9700 5 monclient: authenticate success, global_id 4607 2023-09-30T00:52:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6ddefa9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:52:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6dd77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6ddefa9700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6dd77fe700 1 -- 172.21.15.40:0/2594518573 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd00092e0 con 0x7f6dd813ac20 2023-09-30T00:52:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6ddefa9700 4 set_mon_vals no callback set 2023-09-30T00:52:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6dd77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6dd77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:52:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.884+0000 7f6dd77fe700 10 monclient: dump: 2023-09-30T00:52:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:00.888 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:00.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.885+0000 7f6ddefa9700 10 monclient: shutdown 2023-09-30T00:52:00.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.885+0000 7f6ddefa9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:00.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.885+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2594518573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dd813ac20 msgr2=0x7f6dd813b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:00.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.885+0000 7f6ddefa9700 1 --2- 172.21.15.40:0/2594518573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dd813ac20 0x7f6dd813b080 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6dd0009f20 tx=0x7f6dd0009840 comp rx=0 tx=0).stop 2023-09-30T00:52:00.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.885+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2594518573 shutdown_connections 2023-09-30T00:52:00.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.885+0000 7f6ddefa9700 1 --2- 172.21.15.40:0/2594518573 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dd813ac20 0x7f6dd813b080 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:00.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.885+0000 7f6ddefa9700 1 --2- 172.21.15.40:0/2594518573 >> v2:172.21.15.135:3300/0 conn(0x7f6dd80b1460 0x7f6dd80b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:00.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.885+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2594518573 >> 172.21.15.40:0/2594518573 conn(0x7f6dd80631f0 msgr2=0x7f6dd80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:00.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.886+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2594518573 shutdown_connections 2023-09-30T00:52:00.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.886+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2594518573 wait complete. 2023-09-30T00:52:00.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.887+0000 7f6ddefa9700 10 monclient: build_initial_monmap 2023-09-30T00:52:00.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.887+0000 7f6ddefa9700 10 monclient: monmap: 2023-09-30T00:52:00.892 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:00.892 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:00.892 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:00.893 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:00.893 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:00.893 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:00.893 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:00.893 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:00.893 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:00.893 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:00.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.887+0000 7f6ddefa9700 1 librados: starting msgr at 2023-09-30T00:52:00.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.887+0000 7f6ddefa9700 1 librados: starting objecter 2023-09-30T00:52:00.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.887+0000 7f6ddefa9700 1 Processor -- start 2023-09-30T00:52:00.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.887+0000 7f6ddefa9700 1 -- start start 2023-09-30T00:52:00.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.887+0000 7f6ddefa9700 1 librados: setting wanted keys 2023-09-30T00:52:00.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.887+0000 7f6ddefa9700 1 librados: calling monclient init 2023-09-30T00:52:00.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.887+0000 7f6ddefa9700 10 monclient: init 2023-09-30T00:52:00.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:00.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:52:00.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dd80a5fb0 0x7f6dd813e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:00.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 10 monclient(hunting): picked mon.noname-c con 0x7f6dd80a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:00.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd80b1460 0x7f6dd80a79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:00.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 10 monclient(hunting): picked mon.noname-a con 0x7f6dd80b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:00.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6dd813ac20 0x7f6dd80a7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:00.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 10 monclient(hunting): picked mon.noname-b con 0x7f6dd813ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:00.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:00.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd80ab960 con 0x7f6dd80b1460 2023-09-30T00:52:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd80abc40 con 0x7f6dd813ac20 2023-09-30T00:52:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd80aa4f0 con 0x7f6dd80a5fb0 2023-09-30T00:52:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.888+0000 7f6ddefa9700 10 monclient(hunting): authenticate will time out at 5582.189472s 2023-09-30T00:52:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.889+0000 7f6dd7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd80b1460 0x7f6dd80a79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.889+0000 7f6ddcd45700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dd80a5fb0 0x7f6dd813e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.890+0000 7f6dd7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd80b1460 0x7f6dd80a79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60954/0 (socket says 172.21.15.40:60954) 2023-09-30T00:52:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.890+0000 7f6dd7fff700 1 -- 172.21.15.40:0/2154840029 learned_addr learned my addr 172.21.15.40:0/2154840029 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.890+0000 7f6dd7fff700 10 monclient(hunting): get_auth_request con 0x7f6dd80b1460 auth_method 0 2023-09-30T00:52:00.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.890+0000 7f6dd7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:00.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.890+0000 7f6dd7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:00.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.890+0000 7f6dd7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:00.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.890+0000 7f6ddcd45700 10 monclient(hunting): get_auth_request con 0x7f6dd80a5fb0 auth_method 0 2023-09-30T00:52:00.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.890+0000 7f6ddcd45700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:00.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.890+0000 7f6ddcd45700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:00.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.890+0000 7f6ddcd45700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:00.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.890+0000 7f6dd7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:00.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.890+0000 7f6dd7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:00.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.890+0000 7f6dd7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:00.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.891+0000 7f6ddcd45700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:00.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.891+0000 7f6ddcd45700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:00.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.891+0000 7f6ddcd45700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:00.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.891+0000 7f6ddd546700 1 --2- 172.21.15.40:0/2154840029 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6dd813ac20 0x7f6dd80a7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:00.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.892+0000 7f6ddd546700 10 monclient(hunting): get_auth_request con 0x7f6dd813ac20 auth_method 0 2023-09-30T00:52:00.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.892+0000 7f6ddd546700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:00.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.892+0000 7f6ddd546700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:00.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.892+0000 7f6ddd546700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:00.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.892+0000 7f6ddd546700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:00.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.892+0000 7f6ddd546700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:00.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.892+0000 7f6ddd546700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:00.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.895+0000 7f6ddcd45700 10 monclient(hunting): handle_auth_done global_id 4610 payload 824 2023-09-30T00:52:00.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.895+0000 7f6ddcd45700 1 -- 172.21.15.40:0/2154840029 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6dd813ac20 msgr2=0x7f6dd80a7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:00.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.895+0000 7f6ddcd45700 1 --2- 172.21.15.40:0/2154840029 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6dd813ac20 0x7f6dd80a7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:00.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.895+0000 7f6ddcd45700 1 -- 172.21.15.40:0/2154840029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd80b1460 msgr2=0x7f6dd80a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:00.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.895+0000 7f6ddcd45700 1 --2- 172.21.15.40:0/2154840029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd80b1460 0x7f6dd80a79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:00.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.895+0000 7f6ddcd45700 10 monclient: _finish_hunting 0 2023-09-30T00:52:00.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.895+0000 7f6ddcd45700 1 monclient: found mon.noname-c 2023-09-30T00:52:00.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.895+0000 7f6ddcd45700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:00.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.895+0000 7f6ddcd45700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:52:00.899 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.895+0000 7f6ddcd45700 1 -- 172.21.15.40:0/2154840029 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd0009ae0 con 0x7f6dd80a5fb0 2023-09-30T00:52:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.900+0000 7f6ddcd45700 1 --2- 172.21.15.40:0/2154840029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dd80a5fb0 0x7f6dd813e660 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800ea50 tx=0x7f6dc80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.900+0000 7f6dd4ff9700 10 client.?.objecter ms_handle_connect 0x7f6dd80a5fb0 2023-09-30T00:52:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.900+0000 7f6dd4ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6dd4ff9700 1 -- 172.21.15.40:0/2154840029 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc800ee90 con 0x7f6dd80a5fb0 2023-09-30T00:52:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6dd4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6dd4ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:52:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6dd4ff9700 10 monclient: dump: 2023-09-30T00:52:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:00.903 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:00.904 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:00.904 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:00.904 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:00.904 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:00.904 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:00.904 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:00.904 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:00.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6dd4ff9700 10 monclient: _finish_auth 0 2023-09-30T00:52:00.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6dd4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:00.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6ddefa9700 5 monclient: authenticate success, global_id 4610 2023-09-30T00:52:00.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6ddefa9700 10 monclient: _renew_subs 2023-09-30T00:52:00.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6ddefa9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:00.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2154840029 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd80aa0f0 con 0x7f6dd80a5fb0 2023-09-30T00:52:00.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6ddefa9700 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:00.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6ddefa9700 10 monclient: _renew_subs 2023-09-30T00:52:00.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6ddefa9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:00.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2154840029 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd80aac50 con 0x7f6dd80a5fb0 2023-09-30T00:52:00.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.902+0000 7f6ddefa9700 1 librados: init done 2023-09-30T00:52:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.903+0000 7f6ddefa9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.903+0000 7f6ddefa9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.903+0000 7f6ddefa9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.903+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2154840029 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd81420b0 con 0x7f6dd80a5fb0 2023-09-30T00:52:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.904+0000 7f6dd4ff9700 1 -- 172.21.15.40:0/2154840029 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc8005a00 con 0x7f6dd80a5fb0 2023-09-30T00:52:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.904+0000 7f6dd4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.904+0000 7f6dd4ff9700 1 -- 172.21.15.40:0/2154840029 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc8005b90 con 0x7f6dd80a5fb0 2023-09-30T00:52:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.904+0000 7f6dd4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.904+0000 7f6dd4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:52:00.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.904+0000 7f6dd4ff9700 10 monclient: dump: 2023-09-30T00:52:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:00.907 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:00.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:00.904+0000 7f6dd6ffd700 4 set_mon_vals no callback set 2023-09-30T00:52:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.198+0000 7f6dd4ff9700 1 -- 172.21.15.40:0/2154840029 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6dc800ca00 con 0x7f6dd80a5fb0 2023-09-30T00:52:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.198+0000 7f6dd4ff9700 1 --2- 172.21.15.40:0/2154840029 >> v2:172.21.15.135:6800/116426 conn(0x7f6db8068890 0x7f6db806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.198+0000 7f6dd4ff9700 1 -- 172.21.15.40:0/2154840029 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6dc80846f0 con 0x7f6dd80a5fb0 2023-09-30T00:52:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.198+0000 7f6dd4ff9700 10 client.4610.objecter ms_dispatch 0x7f6dd8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.198+0000 7f6dd4ff9700 3 client.4610.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.198+0000 7f6dd4ff9700 3 client.4610.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.198+0000 7f6dd4ff9700 20 client.4610.objecter dump_active .. 0 homeless 2023-09-30T00:52:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.198+0000 7f6dd4ff9700 1 -- 172.21.15.40:0/2154840029 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6dc8084b40 con 0x7f6dd80a5fb0 2023-09-30T00:52:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.198+0000 7f6dd4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.198+0000 7f6dd4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.198+0000 7f6dd7fff700 1 --2- 172.21.15.40:0/2154840029 >> v2:172.21.15.135:6800/116426 conn(0x7f6db8068890 0x7f6db806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.199+0000 7f6dd7fff700 10 monclient: get_auth_request con 0x7f6db8068890 auth_method 0 2023-09-30T00:52:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.199+0000 7f6dd7fff700 1 --2- 172.21.15.40:0/2154840029 >> v2:172.21.15.135:6800/116426 conn(0x7f6db8068890 0x7f6db806ad40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6dc00078a0 tx=0x7f6dc0007830 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:01.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.199+0000 7f6dd4ff9700 10 client.4610.objecter ms_handle_connect 0x7f6db8068890 2023-09-30T00:52:01.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.451+0000 7f6ddefa9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:01.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.451+0000 7f6ddefa9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:01.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.451+0000 7f6ddefa9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:01.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.451+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2154840029 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6dd8064a00 con 0x7f6dd80a5fb0 2023-09-30T00:52:01.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.457+0000 7f6dd4ff9700 1 -- 172.21.15.40:0/2154840029 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6dc8017100 con 0x7f6dd80a5fb0 2023-09-30T00:52:01.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.457+0000 7f6dd4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:01.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.457+0000 7f6dd4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:01.458 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:01.458 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:01.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 10 librados: watch_flush enter 2023-09-30T00:52:01.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 10 librados: watch_flush exit 2023-09-30T00:52:01.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 20 client.4610.objecter shutdown clearing up homeless session... 2023-09-30T00:52:01.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 10 client.4610.objecter successfully canceled tick 2023-09-30T00:52:01.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2154840029 >> v2:172.21.15.135:6800/116426 conn(0x7f6db8068890 msgr2=0x7f6db806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:01.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 1 --2- 172.21.15.40:0/2154840029 >> v2:172.21.15.135:6800/116426 conn(0x7f6db8068890 0x7f6db806ad40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6dc00078a0 tx=0x7f6dc0007830 comp rx=0 tx=0).stop 2023-09-30T00:52:01.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 10 monclient: shutdown 2023-09-30T00:52:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2154840029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dd80a5fb0 msgr2=0x7f6dd813e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 1 --2- 172.21.15.40:0/2154840029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dd80a5fb0 0x7f6dd813e660 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800ea50 tx=0x7f6dc80091a0 comp rx=0 tx=0).stop 2023-09-30T00:52:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2154840029 shutdown_connections 2023-09-30T00:52:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 1 --2- 172.21.15.40:0/2154840029 >> v2:172.21.15.135:6800/116426 conn(0x7f6db8068890 0x7f6db806ad40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 1 --2- 172.21.15.40:0/2154840029 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6dd813ac20 0x7f6dd80a7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 1 --2- 172.21.15.40:0/2154840029 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6dd80a5fb0 0x7f6dd813e660 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 1 --2- 172.21.15.40:0/2154840029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6dd80b1460 0x7f6dd80a79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2154840029 >> 172.21.15.40:0/2154840029 conn(0x7f6dd80631f0 msgr2=0x7f6dd80afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2154840029 shutdown_connections 2023-09-30T00:52:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 1 -- 172.21.15.40:0/2154840029 wait complete. 2023-09-30T00:52:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:01.460+0000 7f6ddefa9700 1 librados: shutdown 2023-09-30T00:52:04.480 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:04.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.618+0000 7f3ef12f9700 10 monclient: get_monmap_and_config 2023-09-30T00:52:04.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.618+0000 7f3ef12f9700 10 monclient: build_initial_monmap 2023-09-30T00:52:04.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.618+0000 7f3ef12f9700 10 monclient: monmap: 2023-09-30T00:52:04.619 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:04.619 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:04.619 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:04.619355+0000 2023-09-30T00:52:04.619 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:04.619355+0000 2023-09-30T00:52:04.619 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:04.619 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:04.620 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.618+0000 7f3ef12f9700 1 Processor -- start 2023-09-30T00:52:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.619+0000 7f3ef12f9700 1 -- start start 2023-09-30T00:52:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.619+0000 7f3ef12f9700 10 monclient: init 2023-09-30T00:52:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.619+0000 7f3ef12f9700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.619+0000 7f3ef12f9700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-09-30T00:52:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.619+0000 7f3ef12f9700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3eec0a5710 0x7f3eec0a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.619+0000 7f3ef12f9700 10 monclient(hunting): picked mon.noname-a con 0x7f3eec0a5710 addr v2:172.21.15.40:3300/0 2023-09-30T00:52:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.619+0000 7f3ef12f9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3eec1417a0 addr v1:172.21.15.40:6789/0 2023-09-30T00:52:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.619+0000 7f3ef12f9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3eec13afe0 addr v1:172.21.15.135:6789/0 2023-09-30T00:52:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.619+0000 7f3ef12f9700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec0a7fd0 con 0x7f3eec1417a0 2023-09-30T00:52:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.619+0000 7f3ef12f9700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec0a5ae0 con 0x7f3eec13afe0 2023-09-30T00:52:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.619+0000 7f3ef12f9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.619+0000 7f3ef12f9700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eec1422e0 con 0x7f3eec0a5710 2023-09-30T00:52:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.619+0000 7f3ef12f9700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.619+0000 7f3ef12f9700 10 monclient(hunting): authenticate will time out at 5585.920777s 2023-09-30T00:52:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.620+0000 7f3eeaffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3eec0a5710 0x7f3eec0a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.620+0000 7f3eeb7fe700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f3eec13afe0 0x7f3eec13b440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:38074/0 (socket says 172.21.15.40:38074) 2023-09-30T00:52:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.620+0000 7f3eeb7fe700 1 -- 172.21.15.40:0/1399346396 learned_addr learned my addr 172.21.15.40:0/1399346396 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:52:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.621+0000 7f3eeaffd700 10 monclient(hunting): get_auth_request con 0x7f3eec0a5710 auth_method 0 2023-09-30T00:52:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.621+0000 7f3eeaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.621+0000 7f3eeaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.621+0000 7f3eeaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.622+0000 7f3eeaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.622+0000 7f3eeaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.622+0000 7f3eeaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.623+0000 7f3ee9ffb700 1 -- 172.21.15.40:0/1399346396 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279686750 0 0) 0x7f3eec0a5ae0 con 0x7f3eec13afe0 2023-09-30T00:52:04.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.623+0000 7f3ee9ffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.623+0000 7f3ee9ffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.623+0000 7f3ee9ffb700 10 monclient(hunting): my global_id is 4765 2023-09-30T00:52:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.623+0000 7f3ee9ffb700 1 -- 172.21.15.40:0/1399346396 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed4003570 con 0x7f3eec13afe0 2023-09-30T00:52:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.624+0000 7f3eeaffd700 10 monclient(hunting): handle_auth_done global_id 4689 payload 274 2023-09-30T00:52:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.624+0000 7f3eeaffd700 1 -- 172.21.15.40:0/1399346396 >> v1:172.21.15.135:6789/0 conn(0x7f3eec13afe0 legacy=0x7f3eec13b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.624+0000 7f3eeaffd700 1 -- 172.21.15.40:0/1399346396 >> v1:172.21.15.40:6789/0 conn(0x7f3eec1417a0 legacy=0x7f3eec141bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.624+0000 7f3eeaffd700 10 monclient: _finish_hunting 0 2023-09-30T00:52:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.624+0000 7f3eeaffd700 1 monclient: found mon.noname-a 2023-09-30T00:52:04.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.624+0000 7f3eeaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:04.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.624+0000 7f3eeaffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:52:04.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.624+0000 7f3eeaffd700 1 -- 172.21.15.40:0/1399346396 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eec0ab780 con 0x7f3eec0a5710 2023-09-30T00:52:04.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.631+0000 7f3eeaffd700 1 --2- 172.21.15.40:0/1399346396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eec0a5710 0x7f3eec0a75e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3edc0098e0 tx=0x7f3edc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62bcdf1230c05eba server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:04.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ee9ffb700 1 -- 172.21.15.40:0/1399346396 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc00b640 con 0x7f3eec0a5710 2023-09-30T00:52:04.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ee9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:04.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ee9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:52:04.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ee9ffb700 10 monclient: dump: 2023-09-30T00:52:04.637 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:04.637 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:04.637 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:04.637 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:04.637 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:04.638 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:04.638 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:04.638 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:04.638 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:04.638 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:04.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ee9ffb700 10 monclient: _finish_auth 0 2023-09-30T00:52:04.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ee9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:04.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ee9ffb700 1 -- 172.21.15.40:0/1399346396 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3edc00b7a0 con 0x7f3eec0a5710 2023-09-30T00:52:04.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ef12f9700 5 monclient: authenticate success, global_id 4689 2023-09-30T00:52:04.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ef12f9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:52:04.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ee9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:04.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ef12f9700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:04.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ee9ffb700 1 -- 172.21.15.40:0/1399346396 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc00bb40 con 0x7f3eec0a5710 2023-09-30T00:52:04.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ef12f9700 4 set_mon_vals no callback set 2023-09-30T00:52:04.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ee9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:04.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ee9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:52:04.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.636+0000 7f3ee9ffb700 10 monclient: dump: 2023-09-30T00:52:04.639 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:04.639 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:04.639 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:04.639 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:04.639 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:04.639 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:04.639 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:04.639 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:04.640 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:04.640 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:04.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.637+0000 7f3ef12f9700 10 monclient: shutdown 2023-09-30T00:52:04.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.637+0000 7f3ef12f9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:04.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.637+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1399346396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eec0a5710 msgr2=0x7f3eec0a75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:04.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.637+0000 7f3ef12f9700 1 --2- 172.21.15.40:0/1399346396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eec0a5710 0x7f3eec0a75e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3edc0098e0 tx=0x7f3edc0091a0 comp rx=0 tx=0).stop 2023-09-30T00:52:04.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.637+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1399346396 shutdown_connections 2023-09-30T00:52:04.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.637+0000 7f3ef12f9700 1 --2- 172.21.15.40:0/1399346396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eec0a5710 0x7f3eec0a75e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:04.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.637+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1399346396 >> 172.21.15.40:0/1399346396 conn(0x7f3eec069060 msgr2=0x7f3eec0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:04.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.637+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1399346396 shutdown_connections 2023-09-30T00:52:04.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.637+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1399346396 wait complete. 2023-09-30T00:52:04.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.638+0000 7f3ef12f9700 10 monclient: build_initial_monmap 2023-09-30T00:52:04.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.638+0000 7f3ef12f9700 10 monclient: monmap: 2023-09-30T00:52:04.647 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:04.647 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:04.647 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:04.649 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.638+0000 7f3ef12f9700 1 librados: starting msgr at 2023-09-30T00:52:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.638+0000 7f3ef12f9700 1 librados: starting objecter 2023-09-30T00:52:04.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.638+0000 7f3ef12f9700 1 Processor -- start 2023-09-30T00:52:04.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.638+0000 7f3ef12f9700 1 -- start start 2023-09-30T00:52:04.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.638+0000 7f3ef12f9700 1 librados: setting wanted keys 2023-09-30T00:52:04.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.638+0000 7f3ef12f9700 1 librados: calling monclient init 2023-09-30T00:52:04.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.638+0000 7f3ef12f9700 10 monclient: init 2023-09-30T00:52:04.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:04.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:52:04.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eec0a5710 0x7f3eec13ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:04.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 10 monclient(hunting): picked mon.noname-c con 0x7f3eec0a5710 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:04.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3eec13afe0 0x7f3eec13ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:04.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 10 monclient(hunting): picked mon.noname-b con 0x7f3eec13afe0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:04.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eec1417a0 0x7f3eec0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 10 monclient(hunting): picked mon.noname-a con 0x7f3eec1417a0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec13c7b0 con 0x7f3eec1417a0 2023-09-30T00:52:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec13c4e0 con 0x7f3eec13afe0 2023-09-30T00:52:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec0ada80 con 0x7f3eec0a5710 2023-09-30T00:52:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.639+0000 7f3ef12f9700 10 monclient(hunting): authenticate will time out at 5585.940536s 2023-09-30T00:52:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eeaffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eec0a5710 0x7f3eec13ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eea7fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3eec13afe0 0x7f3eec13ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eeb7fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eec1417a0 0x7f3eec0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:04.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eeaffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eec0a5710 0x7f3eec13ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:48654/0 (socket says 172.21.15.40:48654) 2023-09-30T00:52:04.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eea7fc700 10 monclient(hunting): get_auth_request con 0x7f3eec13afe0 auth_method 0 2023-09-30T00:52:04.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eeaffd700 1 -- 172.21.15.40:0/1838214521 learned_addr learned my addr 172.21.15.40:0/1838214521 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:04.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eea7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:04.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eea7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:04.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eea7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:04.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eeaffd700 10 monclient(hunting): get_auth_request con 0x7f3eec0a5710 auth_method 0 2023-09-30T00:52:04.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eeaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:04.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eeaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:04.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eeaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:04.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eeb7fe700 10 monclient(hunting): get_auth_request con 0x7f3eec1417a0 auth_method 0 2023-09-30T00:52:04.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eeb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:04.652 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eeb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:04.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.640+0000 7f3eeb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:04.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.641+0000 7f3eea7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:04.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.641+0000 7f3eea7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:04.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.641+0000 7f3eea7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:04.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.641+0000 7f3eeaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:04.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.641+0000 7f3eeaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:04.653 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.641+0000 7f3eeaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.641+0000 7f3eeb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.641+0000 7f3eeb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.641+0000 7f3eeb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.641+0000 7f3eea7fc700 1 -- 172.21.15.40:0/1838214521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3eec13afe0 msgr2=0x7f3eec13ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-09-30T00:52:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.641+0000 7f3eea7fc700 1 -- 172.21.15.40:0/1838214521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3eec13afe0 msgr2=0x7f3eec13ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T00:52:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.641+0000 7f3eea7fc700 1 --2- 172.21.15.40:0/1838214521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3eec13afe0 0x7f3eec13ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:52:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.641+0000 7f3eea7fc700 1 --2- 172.21.15.40:0/1838214521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3eec13afe0 0x7f3eec13ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:52:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.645+0000 7f3eeb7fe700 10 monclient(hunting): handle_auth_done global_id 4692 payload 824 2023-09-30T00:52:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.645+0000 7f3eeb7fe700 1 -- 172.21.15.40:0/1838214521 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eec0a5710 msgr2=0x7f3eec13ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.645+0000 7f3eeb7fe700 1 --2- 172.21.15.40:0/1838214521 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eec0a5710 0x7f3eec13ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.645+0000 7f3eeb7fe700 1 -- 172.21.15.40:0/1838214521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3eec13afe0 msgr2=0x7f3eec13ea40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T00:52:04.654 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.645+0000 7f3eeb7fe700 1 --2- 172.21.15.40:0/1838214521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3eec13afe0 0x7f3eec13ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.645+0000 7f3eeb7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:52:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.645+0000 7f3eeb7fe700 1 monclient: found mon.noname-a 2023-09-30T00:52:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.645+0000 7f3eeb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.645+0000 7f3eeb7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:52:04.655 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.645+0000 7f3eeb7fe700 1 -- 172.21.15.40:0/1838214521 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc009420 con 0x7f3eec1417a0 2023-09-30T00:52:04.656 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.648+0000 7f3eeb7fe700 1 --2- 172.21.15.40:0/1838214521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eec1417a0 0x7f3eec0abd70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400cbc0 tx=0x7f3ee4001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.648+0000 7f3ed2ffd700 10 client.?.objecter ms_handle_connect 0x7f3eec1417a0 2023-09-30T00:52:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.648+0000 7f3ed2ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ed2ffd700 1 -- 172.21.15.40:0/1838214521 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee400a7e0 con 0x7f3eec1417a0 2023-09-30T00:52:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ed2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ed2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:52:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ed2ffd700 10 monclient: dump: 2023-09-30T00:52:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:04.658 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ed2ffd700 10 monclient: _finish_auth 0 2023-09-30T00:52:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ed2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ed2ffd700 1 -- 172.21.15.40:0/1838214521 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee400a940 con 0x7f3eec1417a0 2023-09-30T00:52:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ef12f9700 5 monclient: authenticate success, global_id 4692 2023-09-30T00:52:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ef12f9700 10 monclient: _renew_subs 2023-09-30T00:52:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ef12f9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1838214521 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eec13ef80 con 0x7f3eec1417a0 2023-09-30T00:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ed2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ed2ffd700 1 -- 172.21.15.40:0/1838214521 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee400acc0 con 0x7f3eec1417a0 2023-09-30T00:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ed2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ed2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ed2ffd700 10 monclient: dump: 2023-09-30T00:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:04.660 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:04.660 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:04.660 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:04.660 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:04.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ee97fa700 4 set_mon_vals no callback set 2023-09-30T00:52:04.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ef12f9700 10 client.4692.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:04.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ef12f9700 10 monclient: _renew_subs 2023-09-30T00:52:04.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ef12f9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:04.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1838214521 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eec0ae6a0 con 0x7f3eec1417a0 2023-09-30T00:52:04.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.652+0000 7f3ef12f9700 1 librados: init done 2023-09-30T00:52:04.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.653+0000 7f3ef12f9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:04.660 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.653+0000 7f3ef12f9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.653+0000 7f3ef12f9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.653+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1838214521 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eec072b90 con 0x7f3eec1417a0 2023-09-30T00:52:04.906 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.906+0000 7f3ed2ffd700 1 -- 172.21.15.40:0/1838214521 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3ee4007300 con 0x7f3eec1417a0 2023-09-30T00:52:04.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.906+0000 7f3ed2ffd700 1 --2- 172.21.15.40:0/1838214521 >> v2:172.21.15.135:6800/116426 conn(0x7f3ed8068960 0x7f3ed806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:04.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.907+0000 7f3eeaffd700 1 --2- 172.21.15.40:0/1838214521 >> v2:172.21.15.135:6800/116426 conn(0x7f3ed8068960 0x7f3ed806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:04.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.907+0000 7f3eeaffd700 10 monclient: get_auth_request con 0x7f3ed8068960 auth_method 0 2023-09-30T00:52:04.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.907+0000 7f3eeaffd700 1 --2- 172.21.15.40:0/1838214521 >> v2:172.21.15.135:6800/116426 conn(0x7f3ed8068960 0x7f3ed806ae10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00c010 tx=0x7f3edc017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:04.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.907+0000 7f3ed2ffd700 10 client.4692.objecter ms_handle_connect 0x7f3ed8068960 2023-09-30T00:52:04.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.912+0000 7f3ed2ffd700 1 -- 172.21.15.40:0/1838214521 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3ee400ae20 con 0x7f3eec1417a0 2023-09-30T00:52:04.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.912+0000 7f3ed2ffd700 10 client.4692.objecter ms_dispatch 0x7f3eec001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:04.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.912+0000 7f3ed2ffd700 3 client.4692.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:04.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.912+0000 7f3ed2ffd700 3 client.4692.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:04.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.912+0000 7f3ed2ffd700 20 client.4692.objecter dump_active .. 0 homeless 2023-09-30T00:52:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.978+0000 7f3ed2ffd700 1 -- 172.21.15.40:0/1838214521 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3ee4052400 con 0x7f3eec1417a0 2023-09-30T00:52:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.978+0000 7f3ed2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:04.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:04.978+0000 7f3ed2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:05.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.234+0000 7f3ef12f9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:05.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.234+0000 7f3ef12f9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:05.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.234+0000 7f3ef12f9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:05.235 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.234+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1838214521 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3eec0a6c00 con 0x7f3eec1417a0 2023-09-30T00:52:05.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.240+0000 7f3ed2ffd700 1 -- 172.21.15.40:0/1838214521 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3ee4055bf0 con 0x7f3eec1417a0 2023-09-30T00:52:05.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.240+0000 7f3ed2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:05.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.240+0000 7f3ed2ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:05.241 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:05.242 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:05.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.242+0000 7f3ef12f9700 10 librados: watch_flush enter 2023-09-30T00:52:05.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 10 librados: watch_flush exit 2023-09-30T00:52:05.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 20 client.4692.objecter shutdown clearing up homeless session... 2023-09-30T00:52:05.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 10 client.4692.objecter successfully canceled tick 2023-09-30T00:52:05.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1838214521 >> v2:172.21.15.135:6800/116426 conn(0x7f3ed8068960 msgr2=0x7f3ed806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:05.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 1 --2- 172.21.15.40:0/1838214521 >> v2:172.21.15.135:6800/116426 conn(0x7f3ed8068960 0x7f3ed806ae10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00c010 tx=0x7f3edc017000 comp rx=0 tx=0).stop 2023-09-30T00:52:05.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 10 monclient: shutdown 2023-09-30T00:52:05.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:05.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1838214521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eec1417a0 msgr2=0x7f3eec0abd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:05.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 1 --2- 172.21.15.40:0/1838214521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eec1417a0 0x7f3eec0abd70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3ee400cbc0 tx=0x7f3ee4001a20 comp rx=0 tx=0).stop 2023-09-30T00:52:05.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1838214521 shutdown_connections 2023-09-30T00:52:05.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 1 --2- 172.21.15.40:0/1838214521 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3eec1417a0 0x7f3eec0abd70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:05.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 1 --2- 172.21.15.40:0/1838214521 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3eec0a5710 0x7f3eec13ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:05.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 1 --2- 172.21.15.40:0/1838214521 >> v2:172.21.15.135:6800/116426 conn(0x7f3ed8068960 0x7f3ed806ae10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:05.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 1 --2- 172.21.15.40:0/1838214521 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3eec13afe0 0x7f3eec13ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:05.245 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1838214521 >> 172.21.15.40:0/1838214521 conn(0x7f3eec069060 msgr2=0x7f3eec0b09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:05.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1838214521 shutdown_connections 2023-09-30T00:52:05.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 1 -- 172.21.15.40:0/1838214521 wait complete. 2023-09-30T00:52:05.246 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:05.243+0000 7f3ef12f9700 1 librados: shutdown 2023-09-30T00:52:08.258 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:08.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.403+0000 7f32146a0700 10 monclient: get_monmap_and_config 2023-09-30T00:52:08.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.403+0000 7f32146a0700 10 monclient: build_initial_monmap 2023-09-30T00:52:08.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.403+0000 7f32146a0700 10 monclient: monmap: 2023-09-30T00:52:08.404 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:08.404 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:08.404 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:08.404424+0000 2023-09-30T00:52:08.404 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:08.404424+0000 2023-09-30T00:52:08.404 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:08.405 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:08.405 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:08.405 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:08.405 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:08.405 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:08.405 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:08.405 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:08.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.403+0000 7f32146a0700 1 Processor -- start 2023-09-30T00:52:08.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 1 -- start start 2023-09-30T00:52:08.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 10 monclient: init 2023-09-30T00:52:08.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:08.406 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-09-30T00:52:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f320c0b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:52:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f320c0b1830 0x7f320c0a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 10 monclient(hunting): picked mon.noname-b con 0x7f320c0b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f320c13ac20 0x7f320c13b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 10 monclient(hunting): picked mon.noname-a con 0x7f320c13ac20 addr v2:172.21.15.40:3300/0 2023-09-30T00:52:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f320c0a5b20 con 0x7f320c0b1460 2023-09-30T00:52:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f320c141e40 con 0x7f320c13ac20 2023-09-30T00:52:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0a64f0 con 0x7f320c0b1830 2023-09-30T00:52:08.407 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:08.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.404+0000 7f32146a0700 10 monclient(hunting): authenticate will time out at 5589.705929s 2023-09-30T00:52:08.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.405+0000 7f3212c3d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f320c13ac20 0x7f320c13b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:08.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.406+0000 7f3212c3d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f320c13ac20 0x7f320c13b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41360/0 (socket says 172.21.15.40:41360) 2023-09-30T00:52:08.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.406+0000 7f3212c3d700 1 -- 172.21.15.40:0/3481195588 learned_addr learned my addr 172.21.15.40:0/3481195588 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:08.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.406+0000 7f3212c3d700 10 monclient(hunting): get_auth_request con 0x7f320c13ac20 auth_method 0 2023-09-30T00:52:08.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.406+0000 7f3212c3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:08.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.406+0000 7f3212c3d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:08.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.406+0000 7f3212c3d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:08.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.407+0000 7f3212c3d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:08.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.407+0000 7f3212c3d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:08.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.407+0000 7f3212c3d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:08.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.410+0000 7f3212c3d700 10 monclient(hunting): handle_auth_done global_id 4695 payload 274 2023-09-30T00:52:08.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.410+0000 7f3212c3d700 1 -- 172.21.15.40:0/3481195588 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f320c0b1830 msgr2=0x7f320c0a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:52:08.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.410+0000 7f3212c3d700 1 --2- 172.21.15.40:0/3481195588 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f320c0b1830 0x7f320c0a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:08.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.410+0000 7f3212c3d700 1 -- 172.21.15.40:0/3481195588 >> v1:172.21.15.40:6789/0 conn(0x7f320c0b1460 legacy=0x7f320c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:08.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.410+0000 7f3212c3d700 10 monclient: _finish_hunting 0 2023-09-30T00:52:08.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.410+0000 7f3212c3d700 1 monclient: found mon.noname-a 2023-09-30T00:52:08.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.410+0000 7f3212c3d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:08.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.410+0000 7f3212c3d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:52:08.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.410+0000 7f3212c3d700 1 -- 172.21.15.40:0/3481195588 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f320c0a71f0 con 0x7f320c13ac20 2023-09-30T00:52:08.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.415+0000 7f3212c3d700 1 --2- 172.21.15.40:0/3481195588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f320c13ac20 0x7f320c13b0a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3204009f50 tx=0x7f3204009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba8c8861d6a9ea8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:08.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f321143a700 1 -- 172.21.15.40:0/3481195588 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320400e070 con 0x7f320c13ac20 2023-09-30T00:52:08.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f321143a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:08.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f321143a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:52:08.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f321143a700 10 monclient: dump: 2023-09-30T00:52:08.422 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:08.422 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:08.422 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:08.423 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:08.423 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:08.423 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:08.423 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:08.423 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:08.423 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:08.423 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:08.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f321143a700 10 monclient: _finish_auth 0 2023-09-30T00:52:08.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f321143a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:08.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f321143a700 1 -- 172.21.15.40:0/3481195588 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320400ba70 con 0x7f320c13ac20 2023-09-30T00:52:08.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f321143a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:08.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f321143a700 1 -- 172.21.15.40:0/3481195588 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32040092e0 con 0x7f320c13ac20 2023-09-30T00:52:08.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f32146a0700 5 monclient: authenticate success, global_id 4695 2023-09-30T00:52:08.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f32146a0700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:08.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f321143a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:08.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f32146a0700 4 set_mon_vals no callback set 2023-09-30T00:52:08.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f321143a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:52:08.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.421+0000 7f321143a700 10 monclient: dump: 2023-09-30T00:52:08.424 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:08.424 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:08.425 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:08.425 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:08.425 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:08.425 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:08.425 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:08.425 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:08.425 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:08.425 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:08.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.422+0000 7f32146a0700 10 monclient: shutdown 2023-09-30T00:52:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.422+0000 7f32146a0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.422+0000 7f32146a0700 1 -- 172.21.15.40:0/3481195588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f320c13ac20 msgr2=0x7f320c13b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.422+0000 7f32146a0700 1 --2- 172.21.15.40:0/3481195588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f320c13ac20 0x7f320c13b0a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3204009f50 tx=0x7f3204009840 comp rx=0 tx=0).stop 2023-09-30T00:52:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.422+0000 7f32146a0700 1 -- 172.21.15.40:0/3481195588 shutdown_connections 2023-09-30T00:52:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.422+0000 7f32146a0700 1 --2- 172.21.15.40:0/3481195588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f320c13ac20 0x7f320c13b0a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.422+0000 7f32146a0700 1 --2- 172.21.15.40:0/3481195588 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f320c0b1830 0x7f320c0a5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.422+0000 7f32146a0700 1 -- 172.21.15.40:0/3481195588 >> 172.21.15.40:0/3481195588 conn(0x7f320c0631f0 msgr2=0x7f320c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.423+0000 7f32146a0700 1 -- 172.21.15.40:0/3481195588 shutdown_connections 2023-09-30T00:52:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.423+0000 7f32146a0700 1 -- 172.21.15.40:0/3481195588 wait complete. 2023-09-30T00:52:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.423+0000 7f32146a0700 10 monclient: build_initial_monmap 2023-09-30T00:52:08.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.423+0000 7f32146a0700 10 monclient: monmap: 2023-09-30T00:52:08.429 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:08.429 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:08.429 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:08.429 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:08.430 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.424+0000 7f32146a0700 1 librados: starting msgr at 2023-09-30T00:52:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.424+0000 7f32146a0700 1 librados: starting objecter 2023-09-30T00:52:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.424+0000 7f32146a0700 1 Processor -- start 2023-09-30T00:52:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.424+0000 7f32146a0700 1 -- start start 2023-09-30T00:52:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.424+0000 7f32146a0700 1 librados: setting wanted keys 2023-09-30T00:52:08.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.424+0000 7f32146a0700 1 librados: calling monclient init 2023-09-30T00:52:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.424+0000 7f32146a0700 10 monclient: init 2023-09-30T00:52:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:52:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f320c0b1460 0x7f320c13e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 10 monclient(hunting): picked mon.noname-c con 0x7f320c0b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f320c0b1830 0x7f320c0a79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 10 monclient(hunting): picked mon.noname-a con 0x7f320c0b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f320c13ac20 0x7f320c0a9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 10 monclient(hunting): picked mon.noname-b con 0x7f320c13ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:08.431 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0ab810 con 0x7f320c0b1830 2023-09-30T00:52:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320c13f3c0 con 0x7f320c13ac20 2023-09-30T00:52:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f320c0a9960 con 0x7f320c0b1460 2023-09-30T00:52:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f32146a0700 10 monclient(hunting): authenticate will time out at 5589.726312s 2023-09-30T00:52:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.425+0000 7f3211c3b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f320c0b1830 0x7f320c0a79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.426+0000 7f3211c3b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f320c0b1830 0x7f320c0a79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:41362/0 (socket says 172.21.15.40:41362) 2023-09-30T00:52:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.426+0000 7f3211c3b700 1 -- 172.21.15.40:0/1776837162 learned_addr learned my addr 172.21.15.40:0/1776837162 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.426+0000 7f3211c3b700 10 monclient(hunting): get_auth_request con 0x7f320c0b1830 auth_method 0 2023-09-30T00:52:08.432 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.426+0000 7f3211c3b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.426+0000 7f3211c3b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.426+0000 7f3211c3b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.426+0000 7f3211c3b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.427+0000 7f3211c3b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:08.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.427+0000 7f3211c3b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.430+0000 7f3211c3b700 10 monclient(hunting): handle_auth_done global_id 4701 payload 824 2023-09-30T00:52:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.430+0000 7f3211c3b700 1 -- 172.21.15.40:0/1776837162 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f320c0b1460 msgr2=0x7f320c13e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.430+0000 7f3211c3b700 1 --2- 172.21.15.40:0/1776837162 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f320c0b1460 0x7f320c13e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.430+0000 7f3211c3b700 1 -- 172.21.15.40:0/1776837162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f320c13ac20 msgr2=0x7f320c0a9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.430+0000 7f3211c3b700 1 --2- 172.21.15.40:0/1776837162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f320c13ac20 0x7f320c0a9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.430+0000 7f3211c3b700 10 monclient: _finish_hunting 0 2023-09-30T00:52:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.430+0000 7f3211c3b700 1 monclient: found mon.noname-a 2023-09-30T00:52:08.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.431+0000 7f3211c3b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.431+0000 7f3211c3b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:52:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.431+0000 7f3211c3b700 1 -- 172.21.15.40:0/1776837162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3204009ae0 con 0x7f320c0b1830 2023-09-30T00:52:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.433+0000 7f3211c3b700 1 --2- 172.21.15.40:0/1776837162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f320c0b1830 0x7f320c0a79f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00b810 tx=0x7f31fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.433+0000 7f32027fc700 10 client.?.objecter ms_handle_connect 0x7f320c0b1830 2023-09-30T00:52:08.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.433+0000 7f32027fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32027fc700 1 -- 172.21.15.40:0/1776837162 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc00f5f0 con 0x7f320c0b1830 2023-09-30T00:52:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32027fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:52:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32027fc700 10 monclient: dump: 2023-09-30T00:52:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:08.439 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:08.440 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:08.440 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:08.440 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:08.440 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:08.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32027fc700 10 monclient: _finish_auth 0 2023-09-30T00:52:08.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:08.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32027fc700 1 -- 172.21.15.40:0/1776837162 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc00f770 con 0x7f320c0b1830 2023-09-30T00:52:08.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32027fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:08.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32027fc700 1 -- 172.21.15.40:0/1776837162 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc00fb30 con 0x7f320c0b1830 2023-09-30T00:52:08.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:08.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32027fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:52:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32027fc700 10 monclient: dump: 2023-09-30T00:52:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:08.441 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:08.442 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32146a0700 5 monclient: authenticate success, global_id 4701 2023-09-30T00:52:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32146a0700 10 monclient: _renew_subs 2023-09-30T00:52:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32146a0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32146a0700 1 -- 172.21.15.40:0/1776837162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320c0aa1c0 con 0x7f320c0b1830 2023-09-30T00:52:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f3210c39700 4 set_mon_vals no callback set 2023-09-30T00:52:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32146a0700 10 client.4701.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32146a0700 10 monclient: _renew_subs 2023-09-30T00:52:08.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32146a0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:08.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32146a0700 1 -- 172.21.15.40:0/1776837162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320c0aa600 con 0x7f320c0b1830 2023-09-30T00:52:08.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.438+0000 7f32146a0700 1 librados: init done 2023-09-30T00:52:08.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.439+0000 7f32146a0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:08.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.439+0000 7f32146a0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:08.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.439+0000 7f32146a0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:08.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.439+0000 7f32146a0700 1 -- 172.21.15.40:0/1776837162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320c0aa820 con 0x7f320c0b1830 2023-09-30T00:52:08.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.470+0000 7f32027fc700 1 -- 172.21.15.40:0/1776837162 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f31fc010480 con 0x7f320c0b1830 2023-09-30T00:52:08.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.470+0000 7f32027fc700 1 --2- 172.21.15.40:0/1776837162 >> v2:172.21.15.135:6800/116426 conn(0x7f31f00689e0 0x7f31f006ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:08.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.470+0000 7f32027fc700 1 -- 172.21.15.40:0/1776837162 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f31fc084930 con 0x7f320c0b1830 2023-09-30T00:52:08.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.470+0000 7f32027fc700 10 client.4701.objecter ms_dispatch 0x7f320c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:08.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.470+0000 7f32027fc700 3 client.4701.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:08.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.470+0000 7f32027fc700 3 client.4701.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:08.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.470+0000 7f32027fc700 20 client.4701.objecter dump_active .. 0 homeless 2023-09-30T00:52:08.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.470+0000 7f321243c700 1 --2- 172.21.15.40:0/1776837162 >> v2:172.21.15.135:6800/116426 conn(0x7f31f00689e0 0x7f31f006ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:08.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.471+0000 7f321243c700 10 monclient: get_auth_request con 0x7f31f00689e0 auth_method 0 2023-09-30T00:52:08.472 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.471+0000 7f32027fc700 10 client.4701.objecter ms_handle_connect 0x7f31f00689e0 2023-09-30T00:52:08.473 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.471+0000 7f321243c700 1 --2- 172.21.15.40:0/1776837162 >> v2:172.21.15.135:6800/116426 conn(0x7f31f00689e0 0x7f31f006ae90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3208001730 tx=0x7f3208007b40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:08.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.752+0000 7f32027fc700 1 -- 172.21.15.40:0/1776837162 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31fc052250 con 0x7f320c0b1830 2023-09-30T00:52:08.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.752+0000 7f32027fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:08.753 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:08.752+0000 7f32027fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.031+0000 7f32146a0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.031+0000 7f32146a0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.031+0000 7f32146a0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.031+0000 7f32146a0700 1 -- 172.21.15.40:0/1776837162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f320c064a00 con 0x7f320c0b1830 2023-09-30T00:52:09.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.038+0000 7f32027fc700 1 -- 172.21.15.40:0/1776837162 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f31fc055870 con 0x7f320c0b1830 2023-09-30T00:52:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.038+0000 7f32027fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.038+0000 7f32027fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:09.039 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:09.039 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.040+0000 7f32146a0700 10 librados: watch_flush enter 2023-09-30T00:52:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.040+0000 7f32146a0700 10 librados: watch_flush exit 2023-09-30T00:52:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 20 client.4701.objecter shutdown clearing up homeless session... 2023-09-30T00:52:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 10 client.4701.objecter successfully canceled tick 2023-09-30T00:52:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 1 -- 172.21.15.40:0/1776837162 >> v2:172.21.15.135:6800/116426 conn(0x7f31f00689e0 msgr2=0x7f31f006ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 1 --2- 172.21.15.40:0/1776837162 >> v2:172.21.15.135:6800/116426 conn(0x7f31f00689e0 0x7f31f006ae90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3208001730 tx=0x7f3208007b40 comp rx=0 tx=0).stop 2023-09-30T00:52:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 10 monclient: shutdown 2023-09-30T00:52:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 1 -- 172.21.15.40:0/1776837162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f320c0b1830 msgr2=0x7f320c0a79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 1 --2- 172.21.15.40:0/1776837162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f320c0b1830 0x7f320c0a79f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f31fc00b810 tx=0x7f31fc009250 comp rx=0 tx=0).stop 2023-09-30T00:52:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 1 -- 172.21.15.40:0/1776837162 shutdown_connections 2023-09-30T00:52:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 1 --2- 172.21.15.40:0/1776837162 >> v2:172.21.15.135:6800/116426 conn(0x7f31f00689e0 0x7f31f006ae90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 1 --2- 172.21.15.40:0/1776837162 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f320c13ac20 0x7f320c0a9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 1 --2- 172.21.15.40:0/1776837162 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f320c0b1460 0x7f320c13e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 1 --2- 172.21.15.40:0/1776837162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f320c0b1830 0x7f320c0a79f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 1 -- 172.21.15.40:0/1776837162 >> 172.21.15.40:0/1776837162 conn(0x7f320c0631f0 msgr2=0x7f320c0635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 1 -- 172.21.15.40:0/1776837162 shutdown_connections 2023-09-30T00:52:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 1 -- 172.21.15.40:0/1776837162 wait complete. 2023-09-30T00:52:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:09.041+0000 7f32146a0700 1 librados: shutdown 2023-09-30T00:52:12.055 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:12.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.196+0000 7f471c6e2700 10 monclient: get_monmap_and_config 2023-09-30T00:52:12.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.196+0000 7f471c6e2700 10 monclient: build_initial_monmap 2023-09-30T00:52:12.197 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.196+0000 7f471c6e2700 10 monclient: monmap: 2023-09-30T00:52:12.197 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:12.197 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:12.197 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:12.197712+0000 2023-09-30T00:52:12.197 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:12.197712+0000 2023-09-30T00:52:12.197 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:12.198 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:12.198 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:12.198 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:12.198 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:12.198 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:12.198 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:12.198 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:12.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.197+0000 7f471c6e2700 1 Processor -- start 2023-09-30T00:52:12.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.197+0000 7f471c6e2700 1 -- start start 2023-09-30T00:52:12.199 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.197+0000 7f471c6e2700 10 monclient: init 2023-09-30T00:52:12.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471c6e2700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:12.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471c6e2700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-09-30T00:52:12.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471c6e2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47140b1460 0x7f47140b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:12.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471c6e2700 10 monclient(hunting): picked mon.noname-b con 0x7f47140b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:12.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471c6e2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47140a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:52:12.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471c6e2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f471413ac20 addr v1:172.21.15.40:6789/0 2023-09-30T00:52:12.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471c6e2700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47140a5b20 con 0x7f471413ac20 2023-09-30T00:52:12.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471c6e2700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47140a71f0 con 0x7f47140a5fb0 2023-09-30T00:52:12.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471c6e2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:12.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471c6e2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47140a74d0 con 0x7f47140b1460 2023-09-30T00:52:12.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471c6e2700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:12.200 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471c6e2700 10 monclient(hunting): authenticate will time out at 5593.499203s 2023-09-30T00:52:12.201 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471a47e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47140b1460 0x7f47140b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:12.201 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471ac7f700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f471413ac20 0x7f471413b0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50622/0 (socket says 172.21.15.40:50622) 2023-09-30T00:52:12.201 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.198+0000 7f471ac7f700 1 -- 172.21.15.40:0/3552601050 learned_addr learned my addr 172.21.15.40:0/3552601050 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:52:12.201 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.199+0000 7f471a47e700 10 monclient(hunting): get_auth_request con 0x7f47140b1460 auth_method 0 2023-09-30T00:52:12.201 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.199+0000 7f471a47e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:12.201 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.199+0000 7f471a47e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:12.201 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.199+0000 7f471a47e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:12.201 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.199+0000 7f471a47e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:12.201 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.199+0000 7f471a47e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:12.202 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.200+0000 7f471a47e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:12.202 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.202+0000 7f471947c700 1 -- 172.21.15.40:0/3552601050 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753180939 0 0) 0x7f47140a71f0 con 0x7f47140a5fb0 2023-09-30T00:52:12.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.202+0000 7f471947c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:12.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.202+0000 7f471947c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:12.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.202+0000 7f471947c700 10 monclient(hunting): my global_id is 4771 2023-09-30T00:52:12.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.202+0000 7f471947c700 1 -- 172.21.15.40:0/3552601050 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4700003570 con 0x7f47140a5fb0 2023-09-30T00:52:12.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.202+0000 7f471a47e700 10 monclient(hunting): handle_auth_done global_id 4616 payload 274 2023-09-30T00:52:12.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.202+0000 7f471a47e700 1 -- 172.21.15.40:0/3552601050 >> v1:172.21.15.135:6789/0 conn(0x7f47140a5fb0 legacy=0x7f4714141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:12.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.202+0000 7f471a47e700 1 -- 172.21.15.40:0/3552601050 >> v1:172.21.15.40:6789/0 conn(0x7f471413ac20 legacy=0x7f471413b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:12.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.202+0000 7f471a47e700 10 monclient: _finish_hunting 0 2023-09-30T00:52:12.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.202+0000 7f471a47e700 1 monclient: found mon.noname-b 2023-09-30T00:52:12.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.202+0000 7f471a47e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:12.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.202+0000 7f471a47e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:52:12.204 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.202+0000 7f471a47e700 1 -- 172.21.15.40:0/3552601050 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47140a63c0 con 0x7f47140b1460 2023-09-30T00:52:12.205 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.205+0000 7f471a47e700 1 --2- 172.21.15.40:0/3552601050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47140b1460 0x7f47140b1830 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4710009910 tx=0x7f47100091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e395942f541669c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:12.207 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.207+0000 7f471947c700 1 -- 172.21.15.40:0/3552601050 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471000b680 con 0x7f47140b1460 2023-09-30T00:52:12.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.207+0000 7f471947c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:12.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.207+0000 7f471947c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:52:12.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.207+0000 7f471947c700 10 monclient: dump: 2023-09-30T00:52:12.208 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:12.208 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:12.208 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:12.208 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:12.208 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:12.208 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:12.208 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:12.208 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:12.208 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:12.208 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:12.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.207+0000 7f471947c700 10 monclient: _finish_auth 0 2023-09-30T00:52:12.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.207+0000 7f471947c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:12.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.207+0000 7f471c6e2700 5 monclient: authenticate success, global_id 4616 2023-09-30T00:52:12.209 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.207+0000 7f471c6e2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:52:12.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.208+0000 7f471947c700 1 -- 172.21.15.40:0/3552601050 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471000b7e0 con 0x7f47140b1460 2023-09-30T00:52:12.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.208+0000 7f471947c700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:12.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.208+0000 7f471947c700 1 -- 172.21.15.40:0/3552601050 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471000bb60 con 0x7f47140b1460 2023-09-30T00:52:12.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.208+0000 7f471c6e2700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:12.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.208+0000 7f471c6e2700 4 set_mon_vals no callback set 2023-09-30T00:52:12.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.208+0000 7f471947c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:12.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.208+0000 7f471947c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:52:12.210 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.208+0000 7f471947c700 10 monclient: dump: 2023-09-30T00:52:12.211 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:12.211 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:12.211 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:12.211 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:12.211 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:12.211 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:12.211 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:12.211 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:12.211 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:12.211 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:12.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.209+0000 7f471c6e2700 10 monclient: shutdown 2023-09-30T00:52:12.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.209+0000 7f471c6e2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:12.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.209+0000 7f471c6e2700 1 -- 172.21.15.40:0/3552601050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47140b1460 msgr2=0x7f47140b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:12.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.209+0000 7f471c6e2700 1 --2- 172.21.15.40:0/3552601050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47140b1460 0x7f47140b1830 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4710009910 tx=0x7f47100091a0 comp rx=0 tx=0).stop 2023-09-30T00:52:12.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.209+0000 7f471c6e2700 1 -- 172.21.15.40:0/3552601050 shutdown_connections 2023-09-30T00:52:12.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.209+0000 7f471c6e2700 1 --2- 172.21.15.40:0/3552601050 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47140b1460 0x7f47140b1830 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:12.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.209+0000 7f471c6e2700 1 -- 172.21.15.40:0/3552601050 >> 172.21.15.40:0/3552601050 conn(0x7f47140631f0 msgr2=0x7f47140635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:12.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.209+0000 7f471c6e2700 1 -- 172.21.15.40:0/3552601050 shutdown_connections 2023-09-30T00:52:12.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.209+0000 7f471c6e2700 1 -- 172.21.15.40:0/3552601050 wait complete. 2023-09-30T00:52:12.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.210+0000 7f471c6e2700 10 monclient: build_initial_monmap 2023-09-30T00:52:12.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.210+0000 7f471c6e2700 10 monclient: monmap: 2023-09-30T00:52:12.214 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:12.214 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:12.214 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:12.214 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:12.214 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:12.215 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.210+0000 7f471c6e2700 1 librados: starting msgr at 2023-09-30T00:52:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.210+0000 7f471c6e2700 1 librados: starting objecter 2023-09-30T00:52:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 1 Processor -- start 2023-09-30T00:52:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 1 -- start start 2023-09-30T00:52:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 1 librados: setting wanted keys 2023-09-30T00:52:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 1 librados: calling monclient init 2023-09-30T00:52:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 10 monclient: init 2023-09-30T00:52:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:12.215 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:52:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47140a5fb0 0x7f471414de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 10 monclient(hunting): picked mon.noname-c con 0x7f47140a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47140b1460 0x7f471413e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 10 monclient(hunting): picked mon.noname-a con 0x7f47140b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f471413ac20 0x7f47140a9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 10 monclient(hunting): picked mon.noname-b con 0x7f471413ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47140a9cb0 con 0x7f47140b1460 2023-09-30T00:52:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47140a9f60 con 0x7f471413ac20 2023-09-30T00:52:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47140abdd0 con 0x7f47140a5fb0 2023-09-30T00:52:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.211+0000 7f471c6e2700 10 monclient(hunting): authenticate will time out at 5593.512791s 2023-09-30T00:52:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.212+0000 7f471a47e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47140a5fb0 0x7f471414de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.212+0000 7f471ac7f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f471413ac20 0x7f47140a9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.212+0000 7f471a47e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47140a5fb0 0x7f471414de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:43922/0 (socket says 172.21.15.40:43922) 2023-09-30T00:52:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.212+0000 7f471a47e700 1 -- 172.21.15.40:0/3707471809 learned_addr learned my addr 172.21.15.40:0/3707471809 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.212+0000 7f471a47e700 10 monclient(hunting): get_auth_request con 0x7f47140a5fb0 auth_method 0 2023-09-30T00:52:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.212+0000 7f471a47e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.212+0000 7f471a47e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.212+0000 7f471a47e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.212+0000 7f4719c7d700 1 --2- 172.21.15.40:0/3707471809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47140b1460 0x7f471413e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.212+0000 7f471ac7f700 10 monclient(hunting): get_auth_request con 0x7f471413ac20 auth_method 0 2023-09-30T00:52:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.212+0000 7f471ac7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.212+0000 7f471ac7f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:12.217 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.212+0000 7f471ac7f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:12.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.213+0000 7f471a47e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:12.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.213+0000 7f471a47e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:12.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.213+0000 7f471a47e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:12.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.213+0000 7f4719c7d700 10 monclient(hunting): get_auth_request con 0x7f47140b1460 auth_method 0 2023-09-30T00:52:12.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.213+0000 7f4719c7d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:12.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.213+0000 7f4719c7d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:12.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.213+0000 7f4719c7d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:12.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.213+0000 7f471ac7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:12.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.213+0000 7f471ac7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:12.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.213+0000 7f471ac7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:12.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.214+0000 7f4719c7d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:12.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.214+0000 7f4719c7d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:12.218 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.214+0000 7f4719c7d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:12.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.216+0000 7f471a47e700 10 monclient(hunting): handle_auth_done global_id 4619 payload 824 2023-09-30T00:52:12.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.216+0000 7f471a47e700 1 -- 172.21.15.40:0/3707471809 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f471413ac20 msgr2=0x7f47140a9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:12.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.216+0000 7f471a47e700 1 --2- 172.21.15.40:0/3707471809 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f471413ac20 0x7f47140a9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:12.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.216+0000 7f471a47e700 1 -- 172.21.15.40:0/3707471809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47140b1460 msgr2=0x7f471413e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:12.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.216+0000 7f471a47e700 1 --2- 172.21.15.40:0/3707471809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47140b1460 0x7f471413e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:12.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.216+0000 7f471a47e700 10 monclient: _finish_hunting 0 2023-09-30T00:52:12.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.216+0000 7f471a47e700 1 monclient: found mon.noname-c 2023-09-30T00:52:12.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.216+0000 7f471a47e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:12.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.217+0000 7f471a47e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:52:12.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.217+0000 7f471a47e700 1 -- 172.21.15.40:0/3707471809 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4710009420 con 0x7f47140a5fb0 2023-09-30T00:52:12.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.219+0000 7f471a47e700 1 --2- 172.21.15.40:0/3707471809 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47140a5fb0 0x7f471414de30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f471000b700 tx=0x7f4710017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:12.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.219+0000 7f470a7fc700 10 client.?.objecter ms_handle_connect 0x7f47140a5fb0 2023-09-30T00:52:12.224 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.219+0000 7f470a7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:12.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f470a7fc700 1 -- 172.21.15.40:0/3707471809 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471000bb60 con 0x7f47140a5fb0 2023-09-30T00:52:12.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f470a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:12.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f470a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:52:12.225 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f470a7fc700 10 monclient: dump: 2023-09-30T00:52:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:12.226 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f470a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:52:12.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f470a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:12.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f470a7fc700 1 -- 172.21.15.40:0/3707471809 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4710000c90 con 0x7f47140a5fb0 2023-09-30T00:52:12.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f471c6e2700 5 monclient: authenticate success, global_id 4619 2023-09-30T00:52:12.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f471c6e2700 10 monclient: _renew_subs 2023-09-30T00:52:12.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f471c6e2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:12.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f471c6e2700 1 -- 172.21.15.40:0/3707471809 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47140a8b30 con 0x7f47140a5fb0 2023-09-30T00:52:12.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f470a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:12.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f470a7fc700 1 -- 172.21.15.40:0/3707471809 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4710027680 con 0x7f47140a5fb0 2023-09-30T00:52:12.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f470a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:12.227 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f470a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:52:12.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f470a7fc700 10 monclient: dump: 2023-09-30T00:52:12.228 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:12.228 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:12.228 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:12.228 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:12.228 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:12.228 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:12.228 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:12.228 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:12.228 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:12.228 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:12.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f4718c7b700 4 set_mon_vals no callback set 2023-09-30T00:52:12.228 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f471c6e2700 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:12.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f471c6e2700 10 monclient: _renew_subs 2023-09-30T00:52:12.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f471c6e2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:12.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f471c6e2700 1 -- 172.21.15.40:0/3707471809 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47140aa5d0 con 0x7f47140a5fb0 2023-09-30T00:52:12.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.224+0000 7f471c6e2700 1 librados: init done 2023-09-30T00:52:12.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.225+0000 7f471c6e2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:12.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.225+0000 7f471c6e2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:12.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.225+0000 7f471c6e2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:12.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.225+0000 7f471c6e2700 1 -- 172.21.15.40:0/3707471809 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4714064a00 con 0x7f47140a5fb0 2023-09-30T00:52:12.519 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.518+0000 7f470a7fc700 1 -- 172.21.15.40:0/3707471809 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f471001a430 con 0x7f47140a5fb0 2023-09-30T00:52:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.518+0000 7f470a7fc700 1 --2- 172.21.15.40:0/3707471809 >> v2:172.21.15.135:6800/116426 conn(0x7f46f8068890 0x7f46f806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.518+0000 7f470a7fc700 1 -- 172.21.15.40:0/3707471809 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f4710085f60 con 0x7f47140a5fb0 2023-09-30T00:52:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.518+0000 7f470a7fc700 10 client.4619.objecter ms_dispatch 0x7f4714001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.518+0000 7f470a7fc700 3 client.4619.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.518+0000 7f470a7fc700 3 client.4619.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:12.520 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.518+0000 7f470a7fc700 20 client.4619.objecter dump_active .. 0 homeless 2023-09-30T00:52:12.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.518+0000 7f470a7fc700 1 -- 172.21.15.40:0/3707471809 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4710086470 con 0x7f47140a5fb0 2023-09-30T00:52:12.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.518+0000 7f470a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:12.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.518+0000 7f470a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:12.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.518+0000 7f4719c7d700 1 --2- 172.21.15.40:0/3707471809 >> v2:172.21.15.135:6800/116426 conn(0x7f46f8068890 0x7f46f806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:12.521 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.519+0000 7f4719c7d700 10 monclient: get_auth_request con 0x7f46f8068890 auth_method 0 2023-09-30T00:52:12.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.519+0000 7f470a7fc700 10 client.4619.objecter ms_handle_connect 0x7f46f8068890 2023-09-30T00:52:12.522 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.519+0000 7f4719c7d700 1 --2- 172.21.15.40:0/3707471809 >> v2:172.21.15.135:6800/116426 conn(0x7f46f8068890 0x7f46f806ad40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f47140ab740 tx=0x7f4704007d50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:12.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.768+0000 7f471c6e2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:12.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.768+0000 7f471c6e2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:12.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.768+0000 7f471c6e2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:12.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.768+0000 7f471c6e2700 1 -- 172.21.15.40:0/3707471809 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f47140a56d0 con 0x7f47140a5fb0 2023-09-30T00:52:12.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.776+0000 7f470a7fc700 1 -- 172.21.15.40:0/3707471809 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4710053880 con 0x7f47140a5fb0 2023-09-30T00:52:12.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.776+0000 7f470a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:12.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.776+0000 7f470a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:12.777 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:12.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.778+0000 7f471c6e2700 10 librados: watch_flush enter 2023-09-30T00:52:12.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.778+0000 7f471c6e2700 10 librados: watch_flush exit 2023-09-30T00:52:12.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.778+0000 7f471c6e2700 20 client.4619.objecter shutdown clearing up homeless session... 2023-09-30T00:52:12.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.778+0000 7f471c6e2700 10 client.4619.objecter successfully canceled tick 2023-09-30T00:52:12.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 1 -- 172.21.15.40:0/3707471809 >> v2:172.21.15.135:6800/116426 conn(0x7f46f8068890 msgr2=0x7f46f806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:12.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 1 --2- 172.21.15.40:0/3707471809 >> v2:172.21.15.135:6800/116426 conn(0x7f46f8068890 0x7f46f806ad40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f47140ab740 tx=0x7f4704007d50 comp rx=0 tx=0).stop 2023-09-30T00:52:12.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 10 monclient: shutdown 2023-09-30T00:52:12.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:12.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 1 -- 172.21.15.40:0/3707471809 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47140a5fb0 msgr2=0x7f471414de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:12.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 1 --2- 172.21.15.40:0/3707471809 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47140a5fb0 0x7f471414de30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f471000b700 tx=0x7f4710017000 comp rx=0 tx=0).stop 2023-09-30T00:52:12.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 1 -- 172.21.15.40:0/3707471809 shutdown_connections 2023-09-30T00:52:12.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 1 --2- 172.21.15.40:0/3707471809 >> v2:172.21.15.135:6800/116426 conn(0x7f46f8068890 0x7f46f806ad40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:12.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 1 --2- 172.21.15.40:0/3707471809 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f471413ac20 0x7f47140a9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:12.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 1 --2- 172.21.15.40:0/3707471809 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f47140a5fb0 0x7f471414de30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:12.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 1 --2- 172.21.15.40:0/3707471809 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47140b1460 0x7f471413e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:12.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 1 -- 172.21.15.40:0/3707471809 >> 172.21.15.40:0/3707471809 conn(0x7f47140631f0 msgr2=0x7f47140b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:12.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 1 -- 172.21.15.40:0/3707471809 shutdown_connections 2023-09-30T00:52:12.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 1 -- 172.21.15.40:0/3707471809 wait complete. 2023-09-30T00:52:12.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:12.779+0000 7f471c6e2700 1 librados: shutdown 2023-09-30T00:52:13.975 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:05:49.687 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:13.979 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:05:49.691 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:14.007 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:05:49.718 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:14.033 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:49.719 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:14.037 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:49.724 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:14.046 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:05:49.758 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:14.048 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:05:49.761 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:14.061 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:05:49.748 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:14.075 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:05:49.787 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:15.797 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:15.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.939+0000 7f10705c5700 10 monclient: get_monmap_and_config 2023-09-30T00:52:15.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.939+0000 7f10705c5700 10 monclient: build_initial_monmap 2023-09-30T00:52:15.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.939+0000 7f10705c5700 10 monclient: monmap: 2023-09-30T00:52:15.940 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:15.940 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:15.940 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:15.940761+0000 2023-09-30T00:52:15.940 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:15.940761+0000 2023-09-30T00:52:15.941 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:15.941 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:15.941 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:15.941 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:15.941 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:15.941 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:15.941 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:15.941 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:15.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.940+0000 7f10705c5700 1 Processor -- start 2023-09-30T00:52:15.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.940+0000 7f10705c5700 1 -- start start 2023-09-30T00:52:15.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.940+0000 7f10705c5700 10 monclient: init 2023-09-30T00:52:15.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f10705c5700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:15.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f10705c5700 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-09-30T00:52:15.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f10705c5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10680b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:52:15.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f10705c5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10680b1830 0x7f10680a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:15.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f10705c5700 10 monclient(hunting): picked mon.noname-b con 0x7f10680b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:15.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f10705c5700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f106813ac20 0x7f106813b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:15.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f10705c5700 10 monclient(hunting): picked mon.noname-c con 0x7f106813ac20 addr v2:172.21.15.135:3300/0 2023-09-30T00:52:15.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f10705c5700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10680a5b20 con 0x7f10680b1460 2023-09-30T00:52:15.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f10705c5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:15.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f10705c5700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1068141e40 con 0x7f106813ac20 2023-09-30T00:52:15.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f10705c5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:15.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f10705c5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10680a64f0 con 0x7f10680b1830 2023-09-30T00:52:15.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f10705c5700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:15.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f10705c5700 10 monclient(hunting): authenticate will time out at 5597.242366s 2023-09-30T00:52:15.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.941+0000 7f106db60700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10680b1830 0x7f10680a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:15.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.942+0000 7f106eb62700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f106813ac20 0x7f106813b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:15.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.942+0000 7f106db60700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10680b1830 0x7f10680a5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:43934/0 (socket says 172.21.15.40:43934) 2023-09-30T00:52:15.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.942+0000 7f106db60700 1 -- 172.21.15.40:0/3362567797 learned_addr learned my addr 172.21.15.40:0/3362567797 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:15.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.942+0000 7f106db60700 10 monclient(hunting): get_auth_request con 0x7f10680b1830 auth_method 0 2023-09-30T00:52:15.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.942+0000 7f106db60700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:15.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.942+0000 7f106db60700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:15.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.942+0000 7f106db60700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:15.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.942+0000 7f106eb62700 10 monclient(hunting): get_auth_request con 0x7f106813ac20 auth_method 0 2023-09-30T00:52:15.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.942+0000 7f106eb62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:15.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.942+0000 7f106eb62700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:15.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.942+0000 7f106eb62700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:15.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.943+0000 7f106db60700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:15.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.943+0000 7f106db60700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:15.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.943+0000 7f106db60700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:15.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.943+0000 7f106eb62700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:15.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.943+0000 7f106eb62700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:15.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.943+0000 7f106eb62700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:15.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.946+0000 7f106db60700 10 monclient(hunting): handle_auth_done global_id 4622 payload 274 2023-09-30T00:52:15.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.946+0000 7f106db60700 1 -- 172.21.15.40:0/3362567797 >> v2:172.21.15.135:3300/0 conn(0x7f106813ac20 msgr2=0x7f106813b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:52:15.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.946+0000 7f106db60700 1 --2- 172.21.15.40:0/3362567797 >> v2:172.21.15.135:3300/0 conn(0x7f106813ac20 0x7f106813b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:15.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.946+0000 7f106db60700 1 -- 172.21.15.40:0/3362567797 >> v1:172.21.15.135:6789/0 conn(0x7f10680b1460 legacy=0x7f1068141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:15.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.946+0000 7f106db60700 10 monclient: _finish_hunting 0 2023-09-30T00:52:15.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.946+0000 7f106db60700 1 monclient: found mon.noname-b 2023-09-30T00:52:15.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.946+0000 7f106db60700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:15.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.946+0000 7f106db60700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:52:15.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.946+0000 7f106db60700 1 -- 172.21.15.40:0/3362567797 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10680a71f0 con 0x7f10680b1830 2023-09-30T00:52:15.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.949+0000 7f106db60700 1 --2- 172.21.15.40:0/3362567797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10680b1830 0x7f10680a5fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f105c0099b0 tx=0x7f105c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed00b970611df143 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:15.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f106d35f700 1 -- 172.21.15.40:0/3362567797 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c009ea0 con 0x7f10680b1830 2023-09-30T00:52:15.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f106d35f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:15.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f106d35f700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:52:15.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f106d35f700 10 monclient: dump: 2023-09-30T00:52:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:15.955 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:15.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f106d35f700 10 monclient: _finish_auth 0 2023-09-30T00:52:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f106d35f700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f106d35f700 1 -- 172.21.15.40:0/3362567797 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f105c00b860 con 0x7f10680b1830 2023-09-30T00:52:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f10705c5700 5 monclient: authenticate success, global_id 4622 2023-09-30T00:52:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f10705c5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:52:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f106d35f700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f10705c5700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f106d35f700 1 -- 172.21.15.40:0/3362567797 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c00bc00 con 0x7f10680b1830 2023-09-30T00:52:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f106d35f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f10705c5700 4 set_mon_vals no callback set 2023-09-30T00:52:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f106d35f700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:52:15.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.954+0000 7f106d35f700 10 monclient: dump: 2023-09-30T00:52:15.957 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:15.957 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:15.957 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:15.957 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:15.957 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:15.957 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:15.957 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:15.957 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:15.957 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:15.957 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:15.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.955+0000 7f10705c5700 10 monclient: shutdown 2023-09-30T00:52:15.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.955+0000 7f10705c5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:15.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.955+0000 7f10705c5700 1 -- 172.21.15.40:0/3362567797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10680b1830 msgr2=0x7f10680a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:15.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.955+0000 7f10705c5700 1 --2- 172.21.15.40:0/3362567797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10680b1830 0x7f10680a5fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f105c0099b0 tx=0x7f105c009250 comp rx=0 tx=0).stop 2023-09-30T00:52:15.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.955+0000 7f10705c5700 1 -- 172.21.15.40:0/3362567797 shutdown_connections 2023-09-30T00:52:15.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.955+0000 7f10705c5700 1 --2- 172.21.15.40:0/3362567797 >> v2:172.21.15.135:3300/0 conn(0x7f106813ac20 0x7f106813b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:15.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.955+0000 7f10705c5700 1 --2- 172.21.15.40:0/3362567797 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f10680b1830 0x7f10680a5fb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:15.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.955+0000 7f10705c5700 1 -- 172.21.15.40:0/3362567797 >> 172.21.15.40:0/3362567797 conn(0x7f10680631f0 msgr2=0x7f10680635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:15.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.955+0000 7f10705c5700 1 -- 172.21.15.40:0/3362567797 shutdown_connections 2023-09-30T00:52:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.955+0000 7f10705c5700 1 -- 172.21.15.40:0/3362567797 wait complete. 2023-09-30T00:52:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.956+0000 7f10705c5700 10 monclient: build_initial_monmap 2023-09-30T00:52:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.956+0000 7f10705c5700 10 monclient: monmap: 2023-09-30T00:52:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:15.963 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:15.964 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.956+0000 7f10705c5700 1 librados: starting msgr at 2023-09-30T00:52:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.956+0000 7f10705c5700 1 librados: starting objecter 2023-09-30T00:52:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.956+0000 7f10705c5700 1 Processor -- start 2023-09-30T00:52:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.956+0000 7f10705c5700 1 -- start start 2023-09-30T00:52:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.956+0000 7f10705c5700 1 librados: setting wanted keys 2023-09-30T00:52:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.956+0000 7f10705c5700 1 librados: calling monclient init 2023-09-30T00:52:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.956+0000 7f10705c5700 10 monclient: init 2023-09-30T00:52:15.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:52:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10680b1460 0x7f106813d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 10 monclient(hunting): picked mon.noname-b con 0x7f10680b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10680b1830 0x7f10680a93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 10 monclient(hunting): picked mon.noname-a con 0x7f10680b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f106813ac20 0x7f1068151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 10 monclient(hunting): picked mon.noname-c con 0x7f106813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10680aaa20 con 0x7f10680b1830 2023-09-30T00:52:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:15.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10680aac90 con 0x7f10680b1460 2023-09-30T00:52:15.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:15.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10680a8630 con 0x7f106813ac20 2023-09-30T00:52:15.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:15.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f10705c5700 10 monclient(hunting): authenticate will time out at 5597.258372s 2023-09-30T00:52:15.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f106e361700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10680b1460 0x7f106813d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:15.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.957+0000 7f106db60700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10680b1830 0x7f10680a93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:15.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.958+0000 7f106e361700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10680b1460 0x7f106813d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:52080/0 (socket says 172.21.15.40:52080) 2023-09-30T00:52:15.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.958+0000 7f106e361700 1 -- 172.21.15.40:0/1491618217 learned_addr learned my addr 172.21.15.40:0/1491618217 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:15.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.958+0000 7f106e361700 10 monclient(hunting): get_auth_request con 0x7f10680b1460 auth_method 0 2023-09-30T00:52:15.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.958+0000 7f106e361700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:15.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.958+0000 7f106e361700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:15.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.958+0000 7f106e361700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:15.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.958+0000 7f106db60700 10 monclient(hunting): get_auth_request con 0x7f10680b1830 auth_method 0 2023-09-30T00:52:15.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.958+0000 7f106db60700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:15.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.958+0000 7f106db60700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:15.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.958+0000 7f106db60700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:15.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.958+0000 7f106e361700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:15.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.958+0000 7f106e361700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:15.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.958+0000 7f106e361700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:15.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.959+0000 7f106eb62700 1 --2- 172.21.15.40:0/1491618217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f106813ac20 0x7f1068151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.959+0000 7f106db60700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.959+0000 7f106db60700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.959+0000 7f106db60700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.959+0000 7f106eb62700 10 monclient(hunting): get_auth_request con 0x7f106813ac20 auth_method 0 2023-09-30T00:52:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.959+0000 7f106eb62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.959+0000 7f106eb62700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.959+0000 7f106eb62700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.960+0000 7f106eb62700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:15.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.960+0000 7f106eb62700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:15.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.960+0000 7f106eb62700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:15.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.962+0000 7f106e361700 10 monclient(hunting): handle_auth_done global_id 4783 payload 824 2023-09-30T00:52:15.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.962+0000 7f106e361700 1 -- 172.21.15.40:0/1491618217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f106813ac20 msgr2=0x7f1068151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.962+0000 7f106e361700 1 --2- 172.21.15.40:0/1491618217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f106813ac20 0x7f1068151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.962+0000 7f106e361700 1 -- 172.21.15.40:0/1491618217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10680b1830 msgr2=0x7f10680a93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.962+0000 7f106e361700 1 --2- 172.21.15.40:0/1491618217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10680b1830 0x7f10680a93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.962+0000 7f106e361700 10 monclient: _finish_hunting 0 2023-09-30T00:52:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.962+0000 7f106e361700 1 monclient: found mon.noname-b 2023-09-30T00:52:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.962+0000 7f106e361700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.962+0000 7f106e361700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:52:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.962+0000 7f106e361700 1 -- 172.21.15.40:0/1491618217 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c0094f0 con 0x7f10680b1460 2023-09-30T00:52:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.964+0000 7f106e361700 1 --2- 172.21.15.40:0/1491618217 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10680b1460 0x7f106813d490 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f105800cb10 tx=0x7f10580017b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.964+0000 7f10667fc700 10 client.?.objecter ms_handle_connect 0x7f10680b1460 2023-09-30T00:52:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.964+0000 7f10667fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10667fc700 1 -- 172.21.15.40:0/1491618217 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800ce20 con 0x7f10680b1460 2023-09-30T00:52:15.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:15.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10667fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:52:15.973 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10667fc700 10 monclient: dump: 2023-09-30T00:52:15.973 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:15.973 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:15.973 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:15.973 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:15.973 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:15.973 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:15.973 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:15.973 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:15.974 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10667fc700 10 monclient: _finish_auth 0 2023-09-30T00:52:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10667fc700 1 -- 172.21.15.40:0/1491618217 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1058006ba0 con 0x7f10680b1460 2023-09-30T00:52:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10705c5700 5 monclient: authenticate success, global_id 4783 2023-09-30T00:52:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10705c5700 10 monclient: _renew_subs 2023-09-30T00:52:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10705c5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10705c5700 1 -- 172.21.15.40:0/1491618217 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10680a8910 con 0x7f10680b1460 2023-09-30T00:52:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10667fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10667fc700 1 -- 172.21.15.40:0/1491618217 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105800a7e0 con 0x7f10680b1460 2023-09-30T00:52:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10667fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:52:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10667fc700 10 monclient: dump: 2023-09-30T00:52:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:15.975 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f106cb5e700 4 set_mon_vals no callback set 2023-09-30T00:52:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10705c5700 10 client.4783.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10705c5700 10 monclient: _renew_subs 2023-09-30T00:52:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10705c5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10705c5700 1 -- 172.21.15.40:0/1491618217 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1068152110 con 0x7f10680b1460 2023-09-30T00:52:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.969+0000 7f10705c5700 1 librados: init done 2023-09-30T00:52:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.970+0000 7f10705c5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.970+0000 7f10705c5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.970+0000 7f10705c5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.970+0000 7f10705c5700 1 -- 172.21.15.40:0/1491618217 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1068064a00 con 0x7f10680b1460 2023-09-30T00:52:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.989+0000 7f10667fc700 1 -- 172.21.15.40:0/1491618217 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f105800abd0 con 0x7f10680b1460 2023-09-30T00:52:15.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.989+0000 7f10667fc700 1 --2- 172.21.15.40:0/1491618217 >> v2:172.21.15.135:6800/116426 conn(0x7f1054068960 0x7f105406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:15.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.990+0000 7f106db60700 1 --2- 172.21.15.40:0/1491618217 >> v2:172.21.15.135:6800/116426 conn(0x7f1054068960 0x7f105406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:15.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.990+0000 7f106db60700 10 monclient: get_auth_request con 0x7f1054068960 auth_method 0 2023-09-30T00:52:15.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.990+0000 7f106db60700 1 --2- 172.21.15.40:0/1491618217 >> v2:172.21.15.135:6800/116426 conn(0x7f1054068960 0x7f105406ae10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f105c009290 tx=0x7f105c017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:15.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.990+0000 7f10667fc700 10 client.4783.objecter ms_handle_connect 0x7f1054068960 2023-09-30T00:52:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.994+0000 7f10667fc700 1 -- 172.21.15.40:0/1491618217 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1058019360 con 0x7f10680b1460 2023-09-30T00:52:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.994+0000 7f10667fc700 10 client.4783.objecter ms_dispatch 0x7f1068001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.994+0000 7f10667fc700 3 client.4783.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.994+0000 7f10667fc700 3 client.4783.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:15.994+0000 7f10667fc700 20 client.4783.objecter dump_active .. 0 homeless 2023-09-30T00:52:16.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.241+0000 7f10667fc700 1 -- 172.21.15.40:0/1491618217 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1058051d10 con 0x7f10680b1460 2023-09-30T00:52:16.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.241+0000 7f10667fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:16.241 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.241+0000 7f10667fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:16.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.507+0000 7f10705c5700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:16.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.507+0000 7f10705c5700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:16.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.507+0000 7f10705c5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:16.508 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.507+0000 7f10705c5700 1 -- 172.21.15.40:0/1491618217 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f10680a74d0 con 0x7f10680b1460 2023-09-30T00:52:16.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.512+0000 7f10667fc700 1 -- 172.21.15.40:0/1491618217 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1058055330 con 0x7f10680b1460 2023-09-30T00:52:16.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.512+0000 7f10667fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:16.512 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.512+0000 7f10667fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:16.513 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:16.513 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:16.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 10 librados: watch_flush enter 2023-09-30T00:52:16.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 10 librados: watch_flush exit 2023-09-30T00:52:16.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 20 client.4783.objecter shutdown clearing up homeless session... 2023-09-30T00:52:16.515 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 10 client.4783.objecter successfully canceled tick 2023-09-30T00:52:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 1 -- 172.21.15.40:0/1491618217 >> v2:172.21.15.135:6800/116426 conn(0x7f1054068960 msgr2=0x7f105406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 1 --2- 172.21.15.40:0/1491618217 >> v2:172.21.15.135:6800/116426 conn(0x7f1054068960 0x7f105406ae10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f105c009290 tx=0x7f105c017000 comp rx=0 tx=0).stop 2023-09-30T00:52:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 10 monclient: shutdown 2023-09-30T00:52:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 1 -- 172.21.15.40:0/1491618217 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10680b1460 msgr2=0x7f106813d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 1 --2- 172.21.15.40:0/1491618217 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10680b1460 0x7f106813d490 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f105800cb10 tx=0x7f10580017b0 comp rx=0 tx=0).stop 2023-09-30T00:52:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 1 -- 172.21.15.40:0/1491618217 shutdown_connections 2023-09-30T00:52:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 1 --2- 172.21.15.40:0/1491618217 >> v2:172.21.15.135:6800/116426 conn(0x7f1054068960 0x7f105406ae10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 1 --2- 172.21.15.40:0/1491618217 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f10680b1460 0x7f106813d490 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 1 --2- 172.21.15.40:0/1491618217 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f106813ac20 0x7f1068151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:16.516 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 1 --2- 172.21.15.40:0/1491618217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10680b1830 0x7f10680a93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 1 -- 172.21.15.40:0/1491618217 >> 172.21.15.40:0/1491618217 conn(0x7f10680631f0 msgr2=0x7f1068139e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 1 -- 172.21.15.40:0/1491618217 shutdown_connections 2023-09-30T00:52:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 1 -- 172.21.15.40:0/1491618217 wait complete. 2023-09-30T00:52:16.517 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:16.514+0000 7f10705c5700 1 librados: shutdown 2023-09-30T00:52:19.531 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:19.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.668+0000 7f733206f700 10 monclient: get_monmap_and_config 2023-09-30T00:52:19.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.668+0000 7f733206f700 10 monclient: build_initial_monmap 2023-09-30T00:52:19.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.668+0000 7f733206f700 10 monclient: monmap: 2023-09-30T00:52:19.669 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:19.669 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:19.669729+0000 2023-09-30T00:52:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:19.669729+0000 2023-09-30T00:52:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:19.670 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:19.670 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:19.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.669+0000 7f733206f700 1 Processor -- start 2023-09-30T00:52:19.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.669+0000 7f733206f700 1 -- start start 2023-09-30T00:52:19.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.669+0000 7f733206f700 10 monclient: init 2023-09-30T00:52:19.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f733206f700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f733206f700 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-09-30T00:52:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f733206f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f732c0b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:52:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f733206f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f732c0b1830 0x7f732c0a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f733206f700 10 monclient(hunting): picked mon.noname-b con 0x7f732c0b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f733206f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f732c13ac20 addr v1:172.21.15.135:6789/0 2023-09-30T00:52:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f733206f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f732c0a5b20 con 0x7f732c0b1460 2023-09-30T00:52:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f733206f700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f732c0a71f0 con 0x7f732c13ac20 2023-09-30T00:52:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f733206f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:19.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f733206f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732c141e40 con 0x7f732c0b1830 2023-09-30T00:52:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f733206f700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f733206f700 10 monclient(hunting): authenticate will time out at 5600.971293s 2023-09-30T00:52:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f732affd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f732c0b1830 0x7f732c0a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f732b7fe700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f732c0b1460 0x7f732c141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50624/0 (socket says 172.21.15.40:50624) 2023-09-30T00:52:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.670+0000 7f732b7fe700 1 -- 172.21.15.40:0/4003954001 learned_addr learned my addr 172.21.15.40:0/4003954001 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:52:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.671+0000 7f732affd700 10 monclient(hunting): get_auth_request con 0x7f732c0b1830 auth_method 0 2023-09-30T00:52:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.671+0000 7f732affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:19.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.671+0000 7f732affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:19.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.671+0000 7f732affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:19.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.672+0000 7f732affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:19.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.672+0000 7f732affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:19.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.672+0000 7f732affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:19.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.673+0000 7f732a7fc700 1 -- 172.21.15.40:0/4003954001 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305326586 0 0) 0x7f732c0a5b20 con 0x7f732c0b1460 2023-09-30T00:52:19.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.673+0000 7f732a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:19.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.673+0000 7f732a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:19.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.673+0000 7f732a7fc700 10 monclient(hunting): my global_id is 4713 2023-09-30T00:52:19.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.673+0000 7f732a7fc700 1 -- 172.21.15.40:0/4003954001 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7318003570 con 0x7f732c0b1460 2023-09-30T00:52:19.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.674+0000 7f732a7fc700 1 -- 172.21.15.40:0/4003954001 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974787715 0 0) 0x7f732c0a71f0 con 0x7f732c13ac20 2023-09-30T00:52:19.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.674+0000 7f732a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.674+0000 7f732a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.674+0000 7f732a7fc700 10 monclient(hunting): my global_id is 4786 2023-09-30T00:52:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.674+0000 7f732a7fc700 1 -- 172.21.15.40:0/4003954001 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f732c0a5b20 con 0x7f732c13ac20 2023-09-30T00:52:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.674+0000 7f732affd700 10 monclient(hunting): handle_auth_done global_id 4628 payload 274 2023-09-30T00:52:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.674+0000 7f732affd700 1 -- 172.21.15.40:0/4003954001 >> v1:172.21.15.135:6789/0 conn(0x7f732c13ac20 legacy=0x7f732c13b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.674+0000 7f732affd700 1 -- 172.21.15.40:0/4003954001 >> v1:172.21.15.40:6789/0 conn(0x7f732c0b1460 legacy=0x7f732c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.674+0000 7f732affd700 10 monclient: _finish_hunting 0 2023-09-30T00:52:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.674+0000 7f732affd700 1 monclient: found mon.noname-b 2023-09-30T00:52:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.674+0000 7f732affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:19.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.674+0000 7f732affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:52:19.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.674+0000 7f732affd700 1 -- 172.21.15.40:0/4003954001 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732c0a74d0 con 0x7f732c0b1830 2023-09-30T00:52:19.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.676+0000 7f732affd700 1 --2- 172.21.15.40:0/4003954001 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f732c0b1830 0x7f732c0a5fb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7314009910 tx=0x7f73140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a4822a522d2ad0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:19.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f732a7fc700 1 -- 172.21.15.40:0/4003954001 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314011630 con 0x7f732c0b1830 2023-09-30T00:52:19.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f732a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:19.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f732a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:52:19.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f732a7fc700 10 monclient: dump: 2023-09-30T00:52:19.682 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:19.682 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:19.682 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:19.682 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:19.682 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:19.682 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:19.682 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:19.683 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:19.683 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:19.683 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:19.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f732a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:52:19.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f732a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:19.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f732a7fc700 1 -- 172.21.15.40:0/4003954001 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7314011790 con 0x7f732c0b1830 2023-09-30T00:52:19.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f732a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:19.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f732a7fc700 1 -- 172.21.15.40:0/4003954001 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314011b30 con 0x7f732c0b1830 2023-09-30T00:52:19.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f732a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:19.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f732a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:52:19.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f732a7fc700 10 monclient: dump: 2023-09-30T00:52:19.683 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:19.684 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:19.684 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:19.684 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:19.684 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:19.684 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:19.684 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:19.684 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:19.684 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:19.684 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:19.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f733206f700 5 monclient: authenticate success, global_id 4628 2023-09-30T00:52:19.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f733206f700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:19.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f733206f700 4 set_mon_vals no callback set 2023-09-30T00:52:19.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f733206f700 10 monclient: shutdown 2023-09-30T00:52:19.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f733206f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:19.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.681+0000 7f733206f700 1 -- 172.21.15.40:0/4003954001 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f732c0b1830 msgr2=0x7f732c0a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:19.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.682+0000 7f733206f700 1 --2- 172.21.15.40:0/4003954001 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f732c0b1830 0x7f732c0a5fb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7314009910 tx=0x7f73140091a0 comp rx=0 tx=0).stop 2023-09-30T00:52:19.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.682+0000 7f733206f700 1 -- 172.21.15.40:0/4003954001 shutdown_connections 2023-09-30T00:52:19.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.682+0000 7f733206f700 1 --2- 172.21.15.40:0/4003954001 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f732c0b1830 0x7f732c0a5fb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:19.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.682+0000 7f733206f700 1 -- 172.21.15.40:0/4003954001 >> 172.21.15.40:0/4003954001 conn(0x7f732c0631f0 msgr2=0x7f732c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:19.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.682+0000 7f733206f700 1 -- 172.21.15.40:0/4003954001 shutdown_connections 2023-09-30T00:52:19.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.682+0000 7f733206f700 1 -- 172.21.15.40:0/4003954001 wait complete. 2023-09-30T00:52:19.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.683+0000 7f733206f700 10 monclient: build_initial_monmap 2023-09-30T00:52:19.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.683+0000 7f733206f700 10 monclient: monmap: 2023-09-30T00:52:19.689 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:19.689 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:19.689 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:19.689 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:19.689 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:19.690 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:19.690 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:19.690 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:19.690 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:19.690 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:19.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.683+0000 7f733206f700 1 librados: starting msgr at 2023-09-30T00:52:19.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.683+0000 7f733206f700 1 librados: starting objecter 2023-09-30T00:52:19.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.683+0000 7f733206f700 1 Processor -- start 2023-09-30T00:52:19.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.683+0000 7f733206f700 1 -- start start 2023-09-30T00:52:19.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.683+0000 7f733206f700 1 librados: setting wanted keys 2023-09-30T00:52:19.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.683+0000 7f733206f700 1 librados: calling monclient init 2023-09-30T00:52:19.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.683+0000 7f733206f700 10 monclient: init 2023-09-30T00:52:19.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:19.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:52:19.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f732c0b1460 0x7f732c13e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:19.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 10 monclient(hunting): picked mon.noname-c con 0x7f732c0b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:19.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f732c0b1830 0x7f732c151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:19.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 10 monclient(hunting): picked mon.noname-a con 0x7f732c0b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:19.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f732c13ac20 0x7f732c0a9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:19.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 10 monclient(hunting): picked mon.noname-b con 0x7f732c13ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:19.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:19.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c151d40 con 0x7f732c0b1830 2023-09-30T00:52:19.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:19.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c151a40 con 0x7f732c13ac20 2023-09-30T00:52:19.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:19.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732c0ab7d0 con 0x7f732c0b1460 2023-09-30T00:52:19.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:19.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.684+0000 7f733206f700 10 monclient(hunting): authenticate will time out at 5600.985592s 2023-09-30T00:52:19.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f732c0b1460 0x7f732c13e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:19.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732bfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f732c13ac20 0x7f732c0a9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:19.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732affd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f732c0b1830 0x7f732c151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:19.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732b7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f732c0b1460 0x7f732c13e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:43954/0 (socket says 172.21.15.40:43954) 2023-09-30T00:52:19.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732b7fe700 1 -- 172.21.15.40:0/1434602508 learned_addr learned my addr 172.21.15.40:0/1434602508 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:19.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732b7fe700 10 monclient(hunting): get_auth_request con 0x7f732c0b1460 auth_method 0 2023-09-30T00:52:19.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:19.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:19.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:19.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732bfff700 10 monclient(hunting): get_auth_request con 0x7f732c13ac20 auth_method 0 2023-09-30T00:52:19.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:19.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:19.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:19.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732affd700 10 monclient(hunting): get_auth_request con 0x7f732c0b1830 auth_method 0 2023-09-30T00:52:19.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:19.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:19.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.685+0000 7f732affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:19.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.686+0000 7f732b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:19.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.686+0000 7f732b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:19.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.686+0000 7f732b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:19.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.686+0000 7f732bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:19.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.686+0000 7f732bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:19.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.686+0000 7f732bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:19.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.686+0000 7f732affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:19.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.686+0000 7f732affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:19.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.686+0000 7f732affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:19.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.690+0000 7f732bfff700 10 monclient(hunting): handle_auth_done global_id 4789 payload 824 2023-09-30T00:52:19.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.690+0000 7f732bfff700 1 -- 172.21.15.40:0/1434602508 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f732c0b1460 msgr2=0x7f732c13e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:19.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.690+0000 7f732bfff700 1 --2- 172.21.15.40:0/1434602508 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f732c0b1460 0x7f732c13e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:19.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.691+0000 7f732bfff700 1 -- 172.21.15.40:0/1434602508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f732c0b1830 msgr2=0x7f732c151500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:19.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.691+0000 7f732bfff700 1 --2- 172.21.15.40:0/1434602508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f732c0b1830 0x7f732c151500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:19.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.691+0000 7f732bfff700 10 monclient: _finish_hunting 0 2023-09-30T00:52:19.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.691+0000 7f732bfff700 1 monclient: found mon.noname-b 2023-09-30T00:52:19.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.691+0000 7f732bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:19.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.691+0000 7f732bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:52:19.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.691+0000 7f732bfff700 1 -- 172.21.15.40:0/1434602508 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7314009420 con 0x7f732c13ac20 2023-09-30T00:52:19.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.693+0000 7f732bfff700 1 --2- 172.21.15.40:0/1434602508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f732c13ac20 0x7f732c0a9270 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f732400c840 tx=0x7f7324002650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:19.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.693+0000 7f73237fe700 10 client.?.objecter ms_handle_connect 0x7f732c13ac20 2023-09-30T00:52:19.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.693+0000 7f73237fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:19.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f73237fe700 1 -- 172.21.15.40:0/1434602508 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324012630 con 0x7f732c13ac20 2023-09-30T00:52:19.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f73237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:19.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f73237fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:52:19.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f73237fe700 10 monclient: dump: 2023-09-30T00:52:19.700 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:19.700 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:19.700 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:19.700 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:19.700 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:19.700 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:19.700 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:19.700 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:19.700 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:19.700 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:19.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f73237fe700 10 monclient: _finish_auth 0 2023-09-30T00:52:19.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f73237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:19.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f73237fe700 1 -- 172.21.15.40:0/1434602508 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7324012790 con 0x7f732c13ac20 2023-09-30T00:52:19.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f733206f700 5 monclient: authenticate success, global_id 4789 2023-09-30T00:52:19.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f733206f700 10 monclient: _renew_subs 2023-09-30T00:52:19.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f733206f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:19.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f733206f700 1 -- 172.21.15.40:0/1434602508 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f732c13ed30 con 0x7f732c13ac20 2023-09-30T00:52:19.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f73237fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:19.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f73237fe700 1 -- 172.21.15.40:0/1434602508 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324012b50 con 0x7f732c13ac20 2023-09-30T00:52:19.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f73237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:19.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f73237fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:52:19.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f73237fe700 10 monclient: dump: 2023-09-30T00:52:19.701 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:19.702 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:19.702 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:19.702 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:19.702 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:19.702 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:19.702 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:19.702 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:19.702 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:19.702 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:19.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f7329ffb700 4 set_mon_vals no callback set 2023-09-30T00:52:19.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f733206f700 10 client.4789.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:19.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f733206f700 10 monclient: _renew_subs 2023-09-30T00:52:19.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f733206f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:19.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f733206f700 1 -- 172.21.15.40:0/1434602508 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f732c0a9970 con 0x7f732c13ac20 2023-09-30T00:52:19.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.698+0000 7f733206f700 1 librados: init done 2023-09-30T00:52:19.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.699+0000 7f733206f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:19.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.699+0000 7f733206f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:19.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.699+0000 7f733206f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:19.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.699+0000 7f733206f700 1 -- 172.21.15.40:0/1434602508 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732c151ef0 con 0x7f732c13ac20 2023-09-30T00:52:19.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.719+0000 7f73237fe700 1 -- 172.21.15.40:0/1434602508 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f732401b400 con 0x7f732c13ac20 2023-09-30T00:52:19.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.720+0000 7f73237fe700 1 --2- 172.21.15.40:0/1434602508 >> v2:172.21.15.135:6800/116426 conn(0x7f730c068960 0x7f730c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:19.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.720+0000 7f732b7fe700 1 --2- 172.21.15.40:0/1434602508 >> v2:172.21.15.135:6800/116426 conn(0x7f730c068960 0x7f730c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:19.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.720+0000 7f732b7fe700 10 monclient: get_auth_request con 0x7f730c068960 auth_method 0 2023-09-30T00:52:19.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.720+0000 7f732b7fe700 1 --2- 172.21.15.40:0/1434602508 >> v2:172.21.15.135:6800/116426 conn(0x7f730c068960 0x7f730c06ae10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f731c0058d0 tx=0x7f731c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:19.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.720+0000 7f73237fe700 10 client.4789.objecter ms_handle_connect 0x7f730c068960 2023-09-30T00:52:19.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.725+0000 7f73237fe700 1 -- 172.21.15.40:0/1434602508 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7324056240 con 0x7f732c13ac20 2023-09-30T00:52:19.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.725+0000 7f73237fe700 10 client.4789.objecter ms_dispatch 0x7f732c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.725+0000 7f73237fe700 3 client.4789.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.725+0000 7f73237fe700 3 client.4789.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:19.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.725+0000 7f73237fe700 20 client.4789.objecter dump_active .. 0 homeless 2023-09-30T00:52:19.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.963+0000 7f73237fe700 1 -- 172.21.15.40:0/1434602508 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7324051e80 con 0x7f732c13ac20 2023-09-30T00:52:19.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.963+0000 7f73237fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:19.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:19.963+0000 7f73237fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:20.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.229+0000 7f733206f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:20.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.229+0000 7f733206f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:20.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.229+0000 7f733206f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:20.230 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.229+0000 7f733206f700 1 -- 172.21.15.40:0/1434602508 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f732c064a00 con 0x7f732c13ac20 2023-09-30T00:52:20.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.234+0000 7f73237fe700 1 -- 172.21.15.40:0/1434602508 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7324016100 con 0x7f732c13ac20 2023-09-30T00:52:20.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.234+0000 7f73237fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:20.234 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.234+0000 7f73237fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:20.234 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:20.234 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:20.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.235+0000 7f733206f700 10 librados: watch_flush enter 2023-09-30T00:52:20.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 10 librados: watch_flush exit 2023-09-30T00:52:20.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 20 client.4789.objecter shutdown clearing up homeless session... 2023-09-30T00:52:20.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 10 client.4789.objecter successfully canceled tick 2023-09-30T00:52:20.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 1 -- 172.21.15.40:0/1434602508 >> v2:172.21.15.135:6800/116426 conn(0x7f730c068960 msgr2=0x7f730c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:20.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 1 --2- 172.21.15.40:0/1434602508 >> v2:172.21.15.135:6800/116426 conn(0x7f730c068960 0x7f730c06ae10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f731c0058d0 tx=0x7f731c009040 comp rx=0 tx=0).stop 2023-09-30T00:52:20.237 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 10 monclient: shutdown 2023-09-30T00:52:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 1 -- 172.21.15.40:0/1434602508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f732c13ac20 msgr2=0x7f732c0a9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 1 --2- 172.21.15.40:0/1434602508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f732c13ac20 0x7f732c0a9270 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f732400c840 tx=0x7f7324002650 comp rx=0 tx=0).stop 2023-09-30T00:52:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 1 -- 172.21.15.40:0/1434602508 shutdown_connections 2023-09-30T00:52:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 1 --2- 172.21.15.40:0/1434602508 >> v2:172.21.15.135:6800/116426 conn(0x7f730c068960 0x7f730c06ae10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 1 --2- 172.21.15.40:0/1434602508 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f732c13ac20 0x7f732c0a9270 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 1 --2- 172.21.15.40:0/1434602508 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f732c0b1460 0x7f732c13e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 1 --2- 172.21.15.40:0/1434602508 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f732c0b1830 0x7f732c151500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 1 -- 172.21.15.40:0/1434602508 >> 172.21.15.40:0/1434602508 conn(0x7f732c0631f0 msgr2=0x7f732c0b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:20.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 1 -- 172.21.15.40:0/1434602508 shutdown_connections 2023-09-30T00:52:20.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 1 -- 172.21.15.40:0/1434602508 wait complete. 2023-09-30T00:52:20.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:20.236+0000 7f733206f700 1 librados: shutdown 2023-09-30T00:52:23.250 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.389+0000 7f3bdc282700 10 monclient: get_monmap_and_config 2023-09-30T00:52:23.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.389+0000 7f3bdc282700 10 monclient: build_initial_monmap 2023-09-30T00:52:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.389+0000 7f3bdc282700 10 monclient: monmap: 2023-09-30T00:52:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:23.390229+0000 2023-09-30T00:52:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:23.390229+0000 2023-09-30T00:52:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:23.391 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:23.391 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:23.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.389+0000 7f3bdc282700 1 Processor -- start 2023-09-30T00:52:23.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.389+0000 7f3bdc282700 1 -- start start 2023-09-30T00:52:23.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.389+0000 7f3bdc282700 10 monclient: init 2023-09-30T00:52:23.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:23.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-09-30T00:52:23.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd40b1460 0x7f3bd40b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:23.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 10 monclient(hunting): picked mon.noname-b con 0x7f3bd40b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:23.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3bd40a5fb0 0x7f3bd4141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:23.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 10 monclient(hunting): picked mon.noname-c con 0x7f3bd40a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:52:23.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3bd40af410 0x7f3bd40af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:23.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 10 monclient(hunting): picked mon.noname-a con 0x7f3bd40af410 addr v2:172.21.15.40:3300/0 2023-09-30T00:52:23.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:23.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd40a71f0 con 0x7f3bd40af410 2023-09-30T00:52:23.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:23.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd40a74d0 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:23.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd40a63c0 con 0x7f3bd40b1460 2023-09-30T00:52:23.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:23.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.390+0000 7f3bdc282700 10 monclient(hunting): authenticate will time out at 5604.691730s 2023-09-30T00:52:23.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.391+0000 7f3bda01e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd40b1460 0x7f3bd40b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:23.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.391+0000 7f3bda81f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3bd40af410 0x7f3bd40af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:23.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.391+0000 7f3bda01e700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd40b1460 0x7f3bd40b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:52436/0 (socket says 172.21.15.40:52436) 2023-09-30T00:52:23.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.391+0000 7f3bda01e700 1 -- 172.21.15.40:0/2700106811 learned_addr learned my addr 172.21.15.40:0/2700106811 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:23.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.391+0000 7f3bda01e700 10 monclient(hunting): get_auth_request con 0x7f3bd40b1460 auth_method 0 2023-09-30T00:52:23.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.391+0000 7f3bda01e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:23.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.391+0000 7f3bda01e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:23.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.391+0000 7f3bda01e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:23.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.391+0000 7f3bd981d700 1 --2- 172.21.15.40:0/2700106811 >> v2:172.21.15.135:3300/0 conn(0x7f3bd40a5fb0 0x7f3bd4141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:23.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.392+0000 7f3bda81f700 10 monclient(hunting): get_auth_request con 0x7f3bd40af410 auth_method 0 2023-09-30T00:52:23.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.392+0000 7f3bda81f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:23.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.392+0000 7f3bda81f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:23.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.392+0000 7f3bda81f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:23.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.392+0000 7f3bd981d700 10 monclient(hunting): get_auth_request con 0x7f3bd40a5fb0 auth_method 0 2023-09-30T00:52:23.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.392+0000 7f3bd981d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:23.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.392+0000 7f3bd981d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:23.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.392+0000 7f3bd981d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:23.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.392+0000 7f3bda01e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:23.397 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.392+0000 7f3bda01e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:23.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.392+0000 7f3bda01e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:23.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.392+0000 7f3bda81f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:23.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.392+0000 7f3bda81f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:23.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.393+0000 7f3bda81f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:23.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.393+0000 7f3bd981d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:23.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.393+0000 7f3bd981d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:23.398 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.393+0000 7f3bd981d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:23.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.397+0000 7f3bd981d700 10 monclient(hunting): handle_auth_done global_id 4792 payload 274 2023-09-30T00:52:23.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.397+0000 7f3bd981d700 1 -- 172.21.15.40:0/2700106811 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd40b1460 msgr2=0x7f3bd40b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:52:23.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.397+0000 7f3bd981d700 1 --2- 172.21.15.40:0/2700106811 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd40b1460 0x7f3bd40b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:23.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.397+0000 7f3bd981d700 1 -- 172.21.15.40:0/2700106811 >> v2:172.21.15.40:3300/0 conn(0x7f3bd40af410 msgr2=0x7f3bd40af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:52:23.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.397+0000 7f3bd981d700 1 --2- 172.21.15.40:0/2700106811 >> v2:172.21.15.40:3300/0 conn(0x7f3bd40af410 0x7f3bd40af890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:23.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.397+0000 7f3bd981d700 10 monclient: _finish_hunting 0 2023-09-30T00:52:23.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.397+0000 7f3bd981d700 1 monclient: found mon.noname-c 2023-09-30T00:52:23.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.397+0000 7f3bd981d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:23.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.397+0000 7f3bd981d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:52:23.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.397+0000 7f3bd981d700 1 -- 172.21.15.40:0/2700106811 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd40a56d0 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.400 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.399+0000 7f3bd981d700 1 --2- 172.21.15.40:0/2700106811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd40a5fb0 0x7f3bd4141730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3bd406f560 tx=0x7f3bc40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8bb35c372a403d0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:23.402 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bd901c700 1 -- 172.21.15.40:0/2700106811 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4011630 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bd901c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:23.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bd901c700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:52:23.403 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bd901c700 10 monclient: dump: 2023-09-30T00:52:23.403 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:23.403 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:23.403 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:23.403 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:23.403 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:23.403 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:23.403 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:23.404 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:23.404 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:23.404 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:23.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bd901c700 10 monclient: _finish_auth 0 2023-09-30T00:52:23.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bd901c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:23.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bd901c700 1 -- 172.21.15.40:0/2700106811 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc40117b0 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bd901c700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:23.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bd901c700 1 -- 172.21.15.40:0/2700106811 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4011b70 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bdc282700 5 monclient: authenticate success, global_id 4792 2023-09-30T00:52:23.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bdc282700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:23.404 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bdc282700 4 set_mon_vals no callback set 2023-09-30T00:52:23.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bd901c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:23.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bd901c700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:52:23.405 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.403+0000 7f3bd901c700 10 monclient: dump: 2023-09-30T00:52:23.405 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:23.405 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:23.405 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:23.405 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:23.405 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:23.405 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:23.405 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:23.405 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:23.406 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:23.406 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:23.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 10 monclient: shutdown 2023-09-30T00:52:23.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:23.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 -- 172.21.15.40:0/2700106811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd40a5fb0 msgr2=0x7f3bd4141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:23.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 --2- 172.21.15.40:0/2700106811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd40a5fb0 0x7f3bd4141730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3bd406f560 tx=0x7f3bc40091a0 comp rx=0 tx=0).stop 2023-09-30T00:52:23.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 -- 172.21.15.40:0/2700106811 shutdown_connections 2023-09-30T00:52:23.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 --2- 172.21.15.40:0/2700106811 >> v2:172.21.15.40:3300/0 conn(0x7f3bd40af410 0x7f3bd40af890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:23.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 --2- 172.21.15.40:0/2700106811 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd40b1460 0x7f3bd40b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:23.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 --2- 172.21.15.40:0/2700106811 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd40a5fb0 0x7f3bd4141730 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:23.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 -- 172.21.15.40:0/2700106811 >> 172.21.15.40:0/2700106811 conn(0x7f3bd40631f0 msgr2=0x7f3bd40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:23.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 -- 172.21.15.40:0/2700106811 shutdown_connections 2023-09-30T00:52:23.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 -- 172.21.15.40:0/2700106811 wait complete. 2023-09-30T00:52:23.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 10 monclient: build_initial_monmap 2023-09-30T00:52:23.410 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 10 monclient: monmap: 2023-09-30T00:52:23.410 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:23.410 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:23.411 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:23.411 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:23.411 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:23.411 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:23.411 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:23.411 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:23.411 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:23.411 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:23.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 librados: starting msgr at 2023-09-30T00:52:23.411 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 librados: starting objecter 2023-09-30T00:52:23.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 Processor -- start 2023-09-30T00:52:23.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 -- start start 2023-09-30T00:52:23.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 librados: setting wanted keys 2023-09-30T00:52:23.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 1 librados: calling monclient init 2023-09-30T00:52:23.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.404+0000 7f3bdc282700 10 monclient: init 2023-09-30T00:52:23.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:23.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:52:23.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd40a5fb0 0x7f3bd40a8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:23.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 10 monclient(hunting): picked mon.noname-b con 0x7f3bd40a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:23.412 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd40af410 0x7f3bd40a9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:23.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 10 monclient(hunting): picked mon.noname-c con 0x7f3bd40af410 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:23.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd40b1460 0x7f3bd40ae120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:23.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 10 monclient(hunting): picked mon.noname-a con 0x7f3bd40b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:23.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:23.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd413e870 con 0x7f3bd40b1460 2023-09-30T00:52:23.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:23.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd40a9d50 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:23.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd413d0d0 con 0x7f3bd40af410 2023-09-30T00:52:23.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:23.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bdc282700 10 monclient(hunting): authenticate will time out at 5604.706152s 2023-09-30T00:52:23.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bd981d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd40af410 0x7f3bd40a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:23.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bda01e700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd40a5fb0 0x7f3bd40a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:23.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.405+0000 7f3bda81f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd40b1460 0x7f3bd40ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:23.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bd981d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd40af410 0x7f3bd40a9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:52450/0 (socket says 172.21.15.40:52450) 2023-09-30T00:52:23.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bd981d700 1 -- 172.21.15.40:0/3382287062 learned_addr learned my addr 172.21.15.40:0/3382287062 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:23.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bd981d700 10 monclient(hunting): get_auth_request con 0x7f3bd40af410 auth_method 0 2023-09-30T00:52:23.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bd981d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:23.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bd981d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:23.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bd981d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:23.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bda01e700 10 monclient(hunting): get_auth_request con 0x7f3bd40a5fb0 auth_method 0 2023-09-30T00:52:23.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bda01e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:23.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bda01e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:23.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bda01e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:23.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bda81f700 10 monclient(hunting): get_auth_request con 0x7f3bd40b1460 auth_method 0 2023-09-30T00:52:23.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bda81f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:23.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bda81f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:23.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bda81f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:23.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bda01e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:23.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bda01e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:23.415 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bda01e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:23.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bd981d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:23.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bd981d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:23.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bd981d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:23.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bda81f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:23.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.406+0000 7f3bda81f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:23.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.407+0000 7f3bda81f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:23.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.411+0000 7f3bda01e700 10 monclient(hunting): handle_auth_done global_id 4795 payload 824 2023-09-30T00:52:23.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.411+0000 7f3bda01e700 1 -- 172.21.15.40:0/3382287062 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd40af410 msgr2=0x7f3bd40a9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:23.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.411+0000 7f3bda01e700 1 --2- 172.21.15.40:0/3382287062 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd40af410 0x7f3bd40a9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:23.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.411+0000 7f3bda01e700 1 -- 172.21.15.40:0/3382287062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd40b1460 msgr2=0x7f3bd40ae120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:23.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.411+0000 7f3bda01e700 1 --2- 172.21.15.40:0/3382287062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd40b1460 0x7f3bd40ae120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:23.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.411+0000 7f3bda01e700 10 monclient: _finish_hunting 0 2023-09-30T00:52:23.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.411+0000 7f3bda01e700 1 monclient: found mon.noname-b 2023-09-30T00:52:23.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.411+0000 7f3bda01e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:23.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.411+0000 7f3bda01e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:52:23.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.411+0000 7f3bda01e700 1 -- 172.21.15.40:0/3382287062 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc4009420 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.413+0000 7f3bda01e700 1 --2- 172.21.15.40:0/3382287062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd40a5fb0 0x7f3bd40a8590 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000e8d0 tx=0x7f3bd0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:23.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.413+0000 7f3bca7fc700 10 client.?.objecter ms_handle_connect 0x7f3bd40a5fb0 2023-09-30T00:52:23.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.413+0000 7f3bca7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:23.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bca7fc700 1 -- 172.21.15.40:0/3382287062 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0013680 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:23.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bca7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:52:23.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bca7fc700 10 monclient: dump: 2023-09-30T00:52:23.420 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:23.420 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:23.420 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:23.420 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:23.420 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:23.421 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:23.421 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:23.421 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:23.421 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:23.421 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:23.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bca7fc700 10 monclient: _finish_auth 0 2023-09-30T00:52:23.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bca7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:23.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bca7fc700 1 -- 172.21.15.40:0/3382287062 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd00137e0 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bdc282700 5 monclient: authenticate success, global_id 4795 2023-09-30T00:52:23.421 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bca7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:23.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bdc282700 10 monclient: _renew_subs 2023-09-30T00:52:23.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bca7fc700 1 -- 172.21.15.40:0/3382287062 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0013b60 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bdc282700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:23.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bdc282700 1 -- 172.21.15.40:0/3382287062 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd40a9690 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bca7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:23.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bca7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:52:23.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bca7fc700 10 monclient: dump: 2023-09-30T00:52:23.422 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:23.422 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:23.422 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:23.423 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:23.423 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:23.423 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:23.423 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:23.423 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:23.423 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:23.423 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:23.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bd881b700 4 set_mon_vals no callback set 2023-09-30T00:52:23.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bdc282700 10 client.4795.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:23.423 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bdc282700 10 monclient: _renew_subs 2023-09-30T00:52:23.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bdc282700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:23.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bdc282700 1 -- 172.21.15.40:0/3382287062 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd40ae660 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.418+0000 7f3bdc282700 1 librados: init done 2023-09-30T00:52:23.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.419+0000 7f3bdc282700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:23.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.419+0000 7f3bdc282700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:23.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.419+0000 7f3bdc282700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:23.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.419+0000 7f3bdc282700 1 -- 172.21.15.40:0/3382287062 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd4064a00 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.671+0000 7f3bca7fc700 1 -- 172.21.15.40:0/3382287062 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3bd00253d0 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.672+0000 7f3bca7fc700 1 --2- 172.21.15.40:0/3382287062 >> v2:172.21.15.135:6800/116426 conn(0x7f3bb8068960 0x7f3bb806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:23.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.672+0000 7f3bd981d700 1 --2- 172.21.15.40:0/3382287062 >> v2:172.21.15.135:6800/116426 conn(0x7f3bb8068960 0x7f3bb806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:23.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.672+0000 7f3bd981d700 10 monclient: get_auth_request con 0x7f3bb8068960 auth_method 0 2023-09-30T00:52:23.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.673+0000 7f3bd981d700 1 --2- 172.21.15.40:0/3382287062 >> v2:172.21.15.135:6800/116426 conn(0x7f3bb8068960 0x7f3bb806ae10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40a8c40 tx=0x7f3bc4017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:23.674 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.673+0000 7f3bca7fc700 10 client.4795.objecter ms_handle_connect 0x7f3bb8068960 2023-09-30T00:52:23.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.677+0000 7f3bca7fc700 1 -- 172.21.15.40:0/3382287062 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3bd0056270 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.677+0000 7f3bca7fc700 10 client.4795.objecter ms_dispatch 0x7f3bd4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:23.676 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.677+0000 7f3bca7fc700 3 client.4795.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:23.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.677+0000 7f3bca7fc700 3 client.4795.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:23.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.677+0000 7f3bca7fc700 20 client.4795.objecter dump_active .. 0 homeless 2023-09-30T00:52:23.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.687+0000 7f3bca7fc700 1 -- 172.21.15.40:0/3382287062 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3bd0051eb0 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.687+0000 7f3bca7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:23.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.687+0000 7f3bca7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.946+0000 7f3bdc282700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.946+0000 7f3bdc282700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.946+0000 7f3bdc282700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:23.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.946+0000 7f3bdc282700 1 -- 172.21.15.40:0/3382287062 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3bd40b4610 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.951+0000 7f3bca7fc700 1 -- 172.21.15.40:0/3382287062 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3bd0021100 con 0x7f3bd40a5fb0 2023-09-30T00:52:23.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.951+0000 7f3bca7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:23.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.951+0000 7f3bca7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:23.952 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:23.952 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:23.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 10 librados: watch_flush enter 2023-09-30T00:52:23.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 10 librados: watch_flush exit 2023-09-30T00:52:23.954 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 20 client.4795.objecter shutdown clearing up homeless session... 2023-09-30T00:52:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 10 client.4795.objecter successfully canceled tick 2023-09-30T00:52:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 1 -- 172.21.15.40:0/3382287062 >> v2:172.21.15.135:6800/116426 conn(0x7f3bb8068960 msgr2=0x7f3bb806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 1 --2- 172.21.15.40:0/3382287062 >> v2:172.21.15.135:6800/116426 conn(0x7f3bb8068960 0x7f3bb806ae10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40a8c40 tx=0x7f3bc4017000 comp rx=0 tx=0).stop 2023-09-30T00:52:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 10 monclient: shutdown 2023-09-30T00:52:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 1 -- 172.21.15.40:0/3382287062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd40a5fb0 msgr2=0x7f3bd40a8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 1 --2- 172.21.15.40:0/3382287062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd40a5fb0 0x7f3bd40a8590 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000e8d0 tx=0x7f3bd0009250 comp rx=0 tx=0).stop 2023-09-30T00:52:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 1 -- 172.21.15.40:0/3382287062 shutdown_connections 2023-09-30T00:52:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 1 --2- 172.21.15.40:0/3382287062 >> v2:172.21.15.135:6800/116426 conn(0x7f3bb8068960 0x7f3bb806ae10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 1 --2- 172.21.15.40:0/3382287062 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3bd40a5fb0 0x7f3bd40a8590 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 1 --2- 172.21.15.40:0/3382287062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3bd40b1460 0x7f3bd40ae120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:23.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 1 --2- 172.21.15.40:0/3382287062 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3bd40af410 0x7f3bd40a9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:23.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 1 -- 172.21.15.40:0/3382287062 >> 172.21.15.40:0/3382287062 conn(0x7f3bd40631f0 msgr2=0x7f3bd40635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:23.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 1 -- 172.21.15.40:0/3382287062 shutdown_connections 2023-09-30T00:52:23.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 1 -- 172.21.15.40:0/3382287062 wait complete. 2023-09-30T00:52:23.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:23.953+0000 7f3bdc282700 1 librados: shutdown 2023-09-30T00:52:26.969 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.110+0000 7fc7f12b2700 10 monclient: get_monmap_and_config 2023-09-30T00:52:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.110+0000 7fc7f12b2700 10 monclient: build_initial_monmap 2023-09-30T00:52:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.110+0000 7fc7f12b2700 10 monclient: monmap: 2023-09-30T00:52:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:27.110822+0000 2023-09-30T00:52:27.110 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:27.110822+0000 2023-09-30T00:52:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:27.111 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.111+0000 7fc7f12b2700 1 Processor -- start 2023-09-30T00:52:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.111+0000 7fc7f12b2700 1 -- start start 2023-09-30T00:52:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.111+0000 7fc7f12b2700 10 monclient: init 2023-09-30T00:52:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7f12b2700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7f12b2700 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-09-30T00:52:27.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7f12b2700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc7ec0b16a0 0x7fc7ec0a7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7f12b2700 10 monclient(hunting): picked mon.noname-a con 0x7fc7ec0b16a0 addr v2:172.21.15.40:3300/0 2023-09-30T00:52:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7f12b2700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc7ec0a54c0 0x7fc7ec0a5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7f12b2700 10 monclient(hunting): picked mon.noname-c con 0x7fc7ec0a54c0 addr v2:172.21.15.135:3300/0 2023-09-30T00:52:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7f12b2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7ec0af400 addr v1:172.21.15.135:6789/0 2023-09-30T00:52:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7f12b2700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ec0a7f00 con 0x7fc7ec0af400 2023-09-30T00:52:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7f12b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7f12b2700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ec0a8390 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7f12b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7f12b2700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ec0b1a70 con 0x7fc7ec0a54c0 2023-09-30T00:52:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7f12b2700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7f12b2700 10 monclient(hunting): authenticate will time out at 5608.412174s 2023-09-30T00:52:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7ea7fc700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc7ec0a54c0 0x7fc7ec0a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.112+0000 7fc7eaffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc7ec0b16a0 0x7fc7ec0a7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.113+0000 7fc7eaffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc7ec0b16a0 0x7fc7ec0a7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48282/0 (socket says 172.21.15.40:48282) 2023-09-30T00:52:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.113+0000 7fc7eaffd700 1 -- 172.21.15.40:0/1812241474 learned_addr learned my addr 172.21.15.40:0/1812241474 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.113+0000 7fc7eaffd700 10 monclient(hunting): get_auth_request con 0x7fc7ec0b16a0 auth_method 0 2023-09-30T00:52:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.113+0000 7fc7eaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.113+0000 7fc7eaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.113+0000 7fc7eaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.113+0000 7fc7ea7fc700 10 monclient(hunting): get_auth_request con 0x7fc7ec0a54c0 auth_method 0 2023-09-30T00:52:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.113+0000 7fc7ea7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.113+0000 7fc7ea7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.113+0000 7fc7ea7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.114+0000 7fc7eaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.114+0000 7fc7eaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.114+0000 7fc7eaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.114+0000 7fc7ea7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.114+0000 7fc7ea7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.114+0000 7fc7ea7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.116+0000 7fc7eaffd700 10 monclient(hunting): handle_auth_done global_id 4725 payload 274 2023-09-30T00:52:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.116+0000 7fc7eaffd700 1 -- 172.21.15.40:0/1812241474 >> v2:172.21.15.135:3300/0 conn(0x7fc7ec0a54c0 msgr2=0x7fc7ec0a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:52:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.116+0000 7fc7eaffd700 1 --2- 172.21.15.40:0/1812241474 >> v2:172.21.15.135:3300/0 conn(0x7fc7ec0a54c0 0x7fc7ec0a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.116+0000 7fc7eaffd700 1 -- 172.21.15.40:0/1812241474 >> v1:172.21.15.135:6789/0 conn(0x7fc7ec0af400 legacy=0x7fc7ec0af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.116+0000 7fc7eaffd700 10 monclient: _finish_hunting 0 2023-09-30T00:52:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.116+0000 7fc7eaffd700 1 monclient: found mon.noname-a 2023-09-30T00:52:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.116+0000 7fc7eaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.116+0000 7fc7eaffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:52:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.116+0000 7fc7eaffd700 1 -- 172.21.15.40:0/1812241474 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ec0a6f30 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.119+0000 7fc7eaffd700 1 --2- 172.21.15.40:0/1812241474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7ec0b16a0 0x7fc7ec0a7630 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec0a7e70 tx=0x7fc7dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d34172ab8ea2373 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7e9ffb700 1 -- 172.21.15.40:0/1812241474 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc009e70 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7e9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:52:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7e9ffb700 10 monclient: dump: 2023-09-30T00:52:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:27.124 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7e9ffb700 10 monclient: _finish_auth 0 2023-09-30T00:52:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7e9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7e9ffb700 1 -- 172.21.15.40:0/1812241474 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7dc00b860 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7f12b2700 5 monclient: authenticate success, global_id 4725 2023-09-30T00:52:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7f12b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:52:27.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7e9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:27.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7f12b2700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:27.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7e9ffb700 1 -- 172.21.15.40:0/1812241474 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc00bc00 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:27.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7f12b2700 4 set_mon_vals no callback set 2023-09-30T00:52:27.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7e9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:52:27.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.123+0000 7fc7e9ffb700 10 monclient: dump: 2023-09-30T00:52:27.126 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:27.126 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:27.126 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:27.126 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:27.126 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:27.126 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:27.127 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.124+0000 7fc7f12b2700 10 monclient: shutdown 2023-09-30T00:52:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.124+0000 7fc7f12b2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.124+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1812241474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7ec0b16a0 msgr2=0x7fc7ec0a7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.124+0000 7fc7f12b2700 1 --2- 172.21.15.40:0/1812241474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7ec0b16a0 0x7fc7ec0a7630 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec0a7e70 tx=0x7fc7dc009250 comp rx=0 tx=0).stop 2023-09-30T00:52:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.124+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1812241474 shutdown_connections 2023-09-30T00:52:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.124+0000 7fc7f12b2700 1 --2- 172.21.15.40:0/1812241474 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7ec0b16a0 0x7fc7ec0a7630 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.124+0000 7fc7f12b2700 1 --2- 172.21.15.40:0/1812241474 >> v2:172.21.15.135:3300/0 conn(0x7fc7ec0a54c0 0x7fc7ec0a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.124+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1812241474 >> 172.21.15.40:0/1812241474 conn(0x7fc7ec06f130 msgr2=0x7fc7ec0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.124+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1812241474 shutdown_connections 2023-09-30T00:52:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.124+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1812241474 wait complete. 2023-09-30T00:52:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.125+0000 7fc7f12b2700 10 monclient: build_initial_monmap 2023-09-30T00:52:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.125+0000 7fc7f12b2700 10 monclient: monmap: 2023-09-30T00:52:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.126+0000 7fc7f12b2700 1 librados: starting msgr at 2023-09-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.126+0000 7fc7f12b2700 1 librados: starting objecter 2023-09-30T00:52:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.126+0000 7fc7f12b2700 1 Processor -- start 2023-09-30T00:52:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.126+0000 7fc7f12b2700 1 -- start start 2023-09-30T00:52:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.126+0000 7fc7f12b2700 1 librados: setting wanted keys 2023-09-30T00:52:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.126+0000 7fc7f12b2700 1 librados: calling monclient init 2023-09-30T00:52:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.126+0000 7fc7f12b2700 10 monclient: init 2023-09-30T00:52:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:52:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7ec0a54c0 0x7fc7ec0ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 10 monclient(hunting): picked mon.noname-a con 0x7fc7ec0a54c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7ec0af400 0x7fc7ec14db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 10 monclient(hunting): picked mon.noname-c con 0x7fc7ec0af400 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7ec0b16a0 0x7fc7ec14e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:27.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 10 monclient(hunting): picked mon.noname-b con 0x7fc7ec0b16a0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:27.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:27.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec0aea60 con 0x7fc7ec0a54c0 2023-09-30T00:52:27.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:27.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec14e5a0 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:27.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec13cf10 con 0x7fc7ec0af400 2023-09-30T00:52:27.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:27.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.127+0000 7fc7f12b2700 10 monclient(hunting): authenticate will time out at 5608.427460s 2023-09-30T00:52:27.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7eaffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7ec0a54c0 0x7fc7ec0ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:27.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7ea7fc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7ec0af400 0x7fc7ec14db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:27.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7eaffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7ec0a54c0 0x7fc7ec0ad0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48296/0 (socket says 172.21.15.40:48296) 2023-09-30T00:52:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7eaffd700 1 -- 172.21.15.40:0/1122902239 learned_addr learned my addr 172.21.15.40:0/1122902239 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7eaffd700 10 monclient(hunting): get_auth_request con 0x7fc7ec0a54c0 auth_method 0 2023-09-30T00:52:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7eaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7eaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7eaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7eb7fe700 1 --2- 172.21.15.40:0/1122902239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7ec0b16a0 0x7fc7ec14e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7ea7fc700 10 monclient(hunting): get_auth_request con 0x7fc7ec0af400 auth_method 0 2023-09-30T00:52:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7ea7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7ea7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7ea7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:27.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7eb7fe700 10 monclient(hunting): get_auth_request con 0x7fc7ec0b16a0 auth_method 0 2023-09-30T00:52:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7eb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7eb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.128+0000 7fc7eb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.129+0000 7fc7eaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.129+0000 7fc7eaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.129+0000 7fc7eaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.129+0000 7fc7eb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.129+0000 7fc7eb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.129+0000 7fc7eb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.129+0000 7fc7ea7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.129+0000 7fc7ea7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:27.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.129+0000 7fc7ea7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:27.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.132+0000 7fc7eb7fe700 10 monclient(hunting): handle_auth_done global_id 4804 payload 824 2023-09-30T00:52:27.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.132+0000 7fc7eb7fe700 1 -- 172.21.15.40:0/1122902239 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7ec0af400 msgr2=0x7fc7ec14db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:27.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.132+0000 7fc7eb7fe700 1 --2- 172.21.15.40:0/1122902239 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7ec0af400 0x7fc7ec14db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:27.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.132+0000 7fc7eb7fe700 1 -- 172.21.15.40:0/1122902239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7ec0a54c0 msgr2=0x7fc7ec0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:27.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.132+0000 7fc7eb7fe700 1 --2- 172.21.15.40:0/1122902239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7ec0a54c0 0x7fc7ec0ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:27.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.132+0000 7fc7eb7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:52:27.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.132+0000 7fc7eb7fe700 1 monclient: found mon.noname-b 2023-09-30T00:52:27.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.132+0000 7fc7eb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:27.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.132+0000 7fc7eb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:52:27.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.132+0000 7fc7eb7fe700 1 -- 172.21.15.40:0/1122902239 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc0094f0 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.135+0000 7fc7eb7fe700 1 --2- 172.21.15.40:0/1122902239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7ec0b16a0 0x7fc7ec14e060 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400c830 tx=0x7fc7e4001690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:27.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.135+0000 7fc7e2ffd700 10 client.?.objecter ms_handle_connect 0x7fc7ec0b16a0 2023-09-30T00:52:27.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.135+0000 7fc7e2ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7e2ffd700 1 -- 172.21.15.40:0/1122902239 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4012630 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7e2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:52:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7e2ffd700 10 monclient: dump: 2023-09-30T00:52:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:27.140 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:27.141 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:27.141 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:27.141 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:27.141 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:27.141 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:27.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7e2ffd700 10 monclient: _finish_auth 0 2023-09-30T00:52:27.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7e2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:27.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7e2ffd700 1 -- 172.21.15.40:0/1122902239 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e4012790 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7e2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7f12b2700 5 monclient: authenticate success, global_id 4804 2023-09-30T00:52:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7e2ffd700 1 -- 172.21.15.40:0/1122902239 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4012b50 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7e2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:52:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7e97fa700 4 set_mon_vals no callback set 2023-09-30T00:52:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7e2ffd700 10 monclient: dump: 2023-09-30T00:52:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:27.142 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:27.143 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:27.143 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:27.143 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:27.143 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:27.143 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:27.143 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:27.143 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:27.143 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:27.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7f12b2700 10 monclient: _renew_subs 2023-09-30T00:52:27.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7f12b2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:27.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1122902239 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ec13edb0 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7f12b2700 10 client.4804.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7f12b2700 10 monclient: _renew_subs 2023-09-30T00:52:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7f12b2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1122902239 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ec0ac560 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.139+0000 7fc7f12b2700 1 librados: init done 2023-09-30T00:52:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.140+0000 7fc7f12b2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.140+0000 7fc7f12b2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.140+0000 7fc7f12b2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:27.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.140+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1122902239 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ec072b90 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.161+0000 7fc7e2ffd700 1 -- 172.21.15.40:0/1122902239 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fc7e4024420 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.161+0000 7fc7e2ffd700 1 --2- 172.21.15.40:0/1122902239 >> v2:172.21.15.135:6800/116426 conn(0x7fc7cc068890 0x7fc7cc06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:27.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.161+0000 7fc7eaffd700 1 --2- 172.21.15.40:0/1122902239 >> v2:172.21.15.135:6800/116426 conn(0x7fc7cc068890 0x7fc7cc06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.161+0000 7fc7eaffd700 10 monclient: get_auth_request con 0x7fc7cc068890 auth_method 0 2023-09-30T00:52:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.162+0000 7fc7e2ffd700 10 client.4804.objecter ms_handle_connect 0x7fc7cc068890 2023-09-30T00:52:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.162+0000 7fc7eaffd700 1 --2- 172.21.15.40:0/1122902239 >> v2:172.21.15.135:6800/116426 conn(0x7fc7cc068890 0x7fc7cc06ad40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc009290 tx=0x7fc7dc017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:27.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.165+0000 7fc7e2ffd700 1 -- 172.21.15.40:0/1122902239 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fc7e40563d0 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.165+0000 7fc7e2ffd700 10 client.4804.objecter ms_dispatch 0x7fc7ec001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:27.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.165+0000 7fc7e2ffd700 3 client.4804.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:27.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.165+0000 7fc7e2ffd700 3 client.4804.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:27.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.165+0000 7fc7e2ffd700 20 client.4804.objecter dump_active .. 0 homeless 2023-09-30T00:52:27.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.408+0000 7fc7e2ffd700 1 -- 172.21.15.40:0/1122902239 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc7e4051e40 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.408+0000 7fc7e2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:27.408 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.408+0000 7fc7e2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:27.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.659+0000 7fc7f12b2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:27.658 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.659+0000 7fc7f12b2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:27.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.659+0000 7fc7f12b2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:27.659 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.659+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1122902239 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc7ec14d770 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.663+0000 7fc7e2ffd700 1 -- 172.21.15.40:0/1122902239 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc7e4055630 con 0x7fc7ec0b16a0 2023-09-30T00:52:27.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.663+0000 7fc7e2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:27.664 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.663+0000 7fc7e2ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:27.665 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:27.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.665+0000 7fc7f12b2700 10 librados: watch_flush enter 2023-09-30T00:52:27.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.665+0000 7fc7f12b2700 10 librados: watch_flush exit 2023-09-30T00:52:27.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.665+0000 7fc7f12b2700 20 client.4804.objecter shutdown clearing up homeless session... 2023-09-30T00:52:27.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.665+0000 7fc7f12b2700 10 client.4804.objecter successfully canceled tick 2023-09-30T00:52:27.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1122902239 >> v2:172.21.15.135:6800/116426 conn(0x7fc7cc068890 msgr2=0x7fc7cc06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:27.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 1 --2- 172.21.15.40:0/1122902239 >> v2:172.21.15.135:6800/116426 conn(0x7fc7cc068890 0x7fc7cc06ad40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc009290 tx=0x7fc7dc017000 comp rx=0 tx=0).stop 2023-09-30T00:52:27.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 10 monclient: shutdown 2023-09-30T00:52:27.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:27.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1122902239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7ec0b16a0 msgr2=0x7fc7ec14e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:27.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 1 --2- 172.21.15.40:0/1122902239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7ec0b16a0 0x7fc7ec14e060 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400c830 tx=0x7fc7e4001690 comp rx=0 tx=0).stop 2023-09-30T00:52:27.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1122902239 shutdown_connections 2023-09-30T00:52:27.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 1 --2- 172.21.15.40:0/1122902239 >> v2:172.21.15.135:6800/116426 conn(0x7fc7cc068890 0x7fc7cc06ad40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:27.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 1 --2- 172.21.15.40:0/1122902239 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc7ec0b16a0 0x7fc7ec14e060 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:27.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 1 --2- 172.21.15.40:0/1122902239 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7ec0a54c0 0x7fc7ec0ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:27.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 1 --2- 172.21.15.40:0/1122902239 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc7ec0af400 0x7fc7ec14db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:27.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1122902239 >> 172.21.15.40:0/1122902239 conn(0x7fc7ec06f130 msgr2=0x7fc7ec0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:27.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1122902239 shutdown_connections 2023-09-30T00:52:27.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 1 -- 172.21.15.40:0/1122902239 wait complete. 2023-09-30T00:52:27.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:27.666+0000 7fc7f12b2700 1 librados: shutdown 2023-09-30T00:52:30.679 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:30.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.823+0000 7f9b7c0db700 10 monclient: get_monmap_and_config 2023-09-30T00:52:30.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.823+0000 7f9b7c0db700 10 monclient: build_initial_monmap 2023-09-30T00:52:30.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.823+0000 7f9b7c0db700 10 monclient: monmap: 2023-09-30T00:52:30.824 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:30.824 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:30.824 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:30.824291+0000 2023-09-30T00:52:30.824 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:30.824291+0000 2023-09-30T00:52:30.824 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:30.824 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:30.824 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:30.824 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:30.824 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:30.824 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:30.824 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:30.825 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:30.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.823+0000 7f9b7c0db700 1 Processor -- start 2023-09-30T00:52:30.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.824+0000 7f9b7c0db700 1 -- start start 2023-09-30T00:52:30.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.824+0000 7f9b7c0db700 10 monclient: init 2023-09-30T00:52:30.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.824+0000 7f9b7c0db700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:30.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.824+0000 7f9b7c0db700 10 monclient: _add_conns ranks=[0,1,3,2,4] 2023-09-30T00:52:30.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.824+0000 7f9b7c0db700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b740b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:52:30.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.824+0000 7f9b7c0db700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b740b1830 addr v1:172.21.15.135:6789/0 2023-09-30T00:52:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.824+0000 7f9b7c0db700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9b740a5fb0 0x7f9b7413b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.824+0000 7f9b7c0db700 10 monclient(hunting): picked mon.noname-a con 0x7f9b740a5fb0 addr v2:172.21.15.40:3300/0 2023-09-30T00:52:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.824+0000 7f9b7c0db700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b740a5b20 con 0x7f9b740b1460 2023-09-30T00:52:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.824+0000 7f9b7c0db700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b740a71f0 con 0x7f9b740b1830 2023-09-30T00:52:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.824+0000 7f9b7c0db700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.824+0000 7f9b7c0db700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b74141e40 con 0x7f9b740a5fb0 2023-09-30T00:52:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.824+0000 7f9b7c0db700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.824+0000 7f9b7c0db700 10 monclient(hunting): authenticate will time out at 5612.125847s 2023-09-30T00:52:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.825+0000 7f9b7a678700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9b740a5fb0 0x7f9b7413b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.825+0000 7f9b79676700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f9b740b1830 0x7f9b7413ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:49798/0 (socket says 172.21.15.40:49798) 2023-09-30T00:52:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.825+0000 7f9b79676700 1 -- 172.21.15.40:0/1423936911 learned_addr learned my addr 172.21.15.40:0/1423936911 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:52:30.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.826+0000 7f9b7a678700 10 monclient(hunting): get_auth_request con 0x7f9b740a5fb0 auth_method 0 2023-09-30T00:52:30.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.826+0000 7f9b7a678700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:30.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.826+0000 7f9b7a678700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:30.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.826+0000 7f9b7a678700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:30.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.826+0000 7f9b7a678700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:30.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.826+0000 7f9b7a678700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:30.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.826+0000 7f9b7a678700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:30.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.829+0000 7f9b78e75700 1 -- 172.21.15.40:0/1423936911 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526702575 0 0) 0x7f9b740a71f0 con 0x7f9b740b1830 2023-09-30T00:52:30.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.829+0000 7f9b78e75700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:30.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.829+0000 7f9b78e75700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:30.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.829+0000 7f9b78e75700 10 monclient(hunting): my global_id is 4807 2023-09-30T00:52:30.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.829+0000 7f9b78e75700 1 -- 172.21.15.40:0/1423936911 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b60003570 con 0x7f9b740b1830 2023-09-30T00:52:30.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.829+0000 7f9b7a678700 10 monclient(hunting): handle_auth_done global_id 4731 payload 274 2023-09-30T00:52:30.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.829+0000 7f9b7a678700 1 -- 172.21.15.40:0/1423936911 >> v1:172.21.15.135:6789/0 conn(0x7f9b740b1830 legacy=0x7f9b7413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:30.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.829+0000 7f9b7a678700 1 -- 172.21.15.40:0/1423936911 >> v1:172.21.15.40:6789/0 conn(0x7f9b740b1460 legacy=0x7f9b74141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:52:30.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.829+0000 7f9b7a678700 10 monclient: _finish_hunting 0 2023-09-30T00:52:30.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.829+0000 7f9b7a678700 1 monclient: found mon.noname-a 2023-09-30T00:52:30.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.829+0000 7f9b7a678700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:30.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.829+0000 7f9b7a678700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:52:30.831 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.829+0000 7f9b7a678700 1 -- 172.21.15.40:0/1423936911 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b740a74d0 con 0x7f9b740a5fb0 2023-09-30T00:52:30.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.834+0000 7f9b7a678700 1 --2- 172.21.15.40:0/1423936911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b740a5fb0 0x7f9b7413b330 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009e90 tx=0x7f9b6c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f77fcf43f210094 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:30.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b78e75700 1 -- 172.21.15.40:0/1423936911 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c00e070 con 0x7f9b740a5fb0 2023-09-30T00:52:30.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b78e75700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:30.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b78e75700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:52:30.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b78e75700 10 monclient: dump: 2023-09-30T00:52:30.841 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:30.841 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:30.841 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:30.841 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:30.841 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:30.841 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:30.841 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:30.842 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:30.842 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:30.842 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:30.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b78e75700 10 monclient: _finish_auth 0 2023-09-30T00:52:30.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b78e75700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:30.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b7c0db700 5 monclient: authenticate success, global_id 4731 2023-09-30T00:52:30.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b78e75700 1 -- 172.21.15.40:0/1423936911 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6c00bbb0 con 0x7f9b740a5fb0 2023-09-30T00:52:30.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b78e75700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:30.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b7c0db700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:30.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b7c0db700 4 set_mon_vals no callback set 2023-09-30T00:52:30.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b78e75700 1 -- 172.21.15.40:0/1423936911 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c012430 con 0x7f9b740a5fb0 2023-09-30T00:52:30.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b78e75700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:30.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b78e75700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:52:30.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.840+0000 7f9b78e75700 10 monclient: dump: 2023-09-30T00:52:30.843 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:30.843 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:30.843 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:30.843 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:30.844 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:30.844 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:30.844 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:30.844 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:30.844 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:30.844 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:30.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.841+0000 7f9b7c0db700 10 monclient: shutdown 2023-09-30T00:52:30.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.841+0000 7f9b7c0db700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:30.847 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.841+0000 7f9b7c0db700 1 -- 172.21.15.40:0/1423936911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b740a5fb0 msgr2=0x7f9b7413b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:30.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.841+0000 7f9b7c0db700 1 --2- 172.21.15.40:0/1423936911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b740a5fb0 0x7f9b7413b330 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c009e90 tx=0x7f9b6c009780 comp rx=0 tx=0).stop 2023-09-30T00:52:30.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.841+0000 7f9b7c0db700 1 -- 172.21.15.40:0/1423936911 shutdown_connections 2023-09-30T00:52:30.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.841+0000 7f9b7c0db700 1 --2- 172.21.15.40:0/1423936911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b740a5fb0 0x7f9b7413b330 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:30.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.841+0000 7f9b7c0db700 1 -- 172.21.15.40:0/1423936911 >> 172.21.15.40:0/1423936911 conn(0x7f9b740631f0 msgr2=0x7f9b740635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:30.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.841+0000 7f9b7c0db700 1 -- 172.21.15.40:0/1423936911 shutdown_connections 2023-09-30T00:52:30.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.841+0000 7f9b7c0db700 1 -- 172.21.15.40:0/1423936911 wait complete. 2023-09-30T00:52:30.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.842+0000 7f9b7c0db700 10 monclient: build_initial_monmap 2023-09-30T00:52:30.848 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.842+0000 7f9b7c0db700 10 monclient: monmap: 2023-09-30T00:52:30.848 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:30.848 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:30.848 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:30.848 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:30.849 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:30.849 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:30.849 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:30.849 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:30.849 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:30.849 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:30.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.842+0000 7f9b7c0db700 1 librados: starting msgr at 2023-09-30T00:52:30.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.842+0000 7f9b7c0db700 1 librados: starting objecter 2023-09-30T00:52:30.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.843+0000 7f9b7c0db700 1 Processor -- start 2023-09-30T00:52:30.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.843+0000 7f9b7c0db700 1 -- start start 2023-09-30T00:52:30.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.843+0000 7f9b7c0db700 1 librados: setting wanted keys 2023-09-30T00:52:30.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.843+0000 7f9b7c0db700 1 librados: calling monclient init 2023-09-30T00:52:30.849 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.843+0000 7f9b7c0db700 10 monclient: init 2023-09-30T00:52:30.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.843+0000 7f9b7c0db700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:30.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.843+0000 7f9b7c0db700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:52:30.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.843+0000 7f9b7c0db700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b740a5fb0 0x7f9b7413e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:30.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b7c0db700 10 monclient(hunting): picked mon.noname-a con 0x7f9b740a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:30.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b7c0db700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b740b1460 0x7f9b74151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:30.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b7c0db700 10 monclient(hunting): picked mon.noname-c con 0x7f9b740b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:30.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b7c0db700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b740b1830 0x7f9b740a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:30.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b7c0db700 10 monclient(hunting): picked mon.noname-b con 0x7f9b740b1830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:30.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b7c0db700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:30.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b7c0db700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b740abee0 con 0x7f9b740a5fb0 2023-09-30T00:52:30.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b7c0db700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:30.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b7c0db700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b74151e60 con 0x7f9b740b1830 2023-09-30T00:52:30.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b7c0db700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:30.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b7c0db700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b74151aa0 con 0x7f9b740b1460 2023-09-30T00:52:30.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b7c0db700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:30.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b7c0db700 10 monclient(hunting): authenticate will time out at 5612.145098s 2023-09-30T00:52:30.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b79e77700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b740a5fb0 0x7f9b7413e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:30.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b79676700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b740b1460 0x7f9b74151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:30.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.844+0000 7f9b7a678700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b740b1830 0x7f9b740a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:30.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79e77700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b740a5fb0 0x7f9b7413e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48314/0 (socket says 172.21.15.40:48314) 2023-09-30T00:52:30.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79e77700 1 -- 172.21.15.40:0/4054284686 learned_addr learned my addr 172.21.15.40:0/4054284686 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:30.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79e77700 10 monclient(hunting): get_auth_request con 0x7f9b740a5fb0 auth_method 0 2023-09-30T00:52:30.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79e77700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:30.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79e77700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:30.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79e77700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:30.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79676700 10 monclient(hunting): get_auth_request con 0x7f9b740b1460 auth_method 0 2023-09-30T00:52:30.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79676700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:30.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79676700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:30.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79676700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:30.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b7a678700 10 monclient(hunting): get_auth_request con 0x7f9b740b1830 auth_method 0 2023-09-30T00:52:30.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b7a678700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:30.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b7a678700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:30.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b7a678700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:30.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79e77700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:30.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79e77700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:30.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79e77700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:30.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79676700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:30.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79676700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:30.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.845+0000 7f9b79676700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:30.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.846+0000 7f9b7a678700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:30.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.846+0000 7f9b7a678700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:30.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.846+0000 7f9b7a678700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:30.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.849+0000 7f9b79e77700 10 monclient(hunting): handle_auth_done global_id 4734 payload 824 2023-09-30T00:52:30.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.849+0000 7f9b79e77700 1 -- 172.21.15.40:0/4054284686 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b740b1460 msgr2=0x7f9b74151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:30.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.849+0000 7f9b79e77700 1 --2- 172.21.15.40:0/4054284686 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b740b1460 0x7f9b74151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:30.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.849+0000 7f9b79e77700 1 -- 172.21.15.40:0/4054284686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b740b1830 msgr2=0x7f9b740a99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:30.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.849+0000 7f9b79e77700 1 --2- 172.21.15.40:0/4054284686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b740b1830 0x7f9b740a99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:30.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.849+0000 7f9b79e77700 10 monclient: _finish_hunting 0 2023-09-30T00:52:30.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.849+0000 7f9b79e77700 1 monclient: found mon.noname-a 2023-09-30T00:52:30.854 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.849+0000 7f9b79e77700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:30.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.849+0000 7f9b79e77700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:52:30.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.849+0000 7f9b79e77700 1 -- 172.21.15.40:0/4054284686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c009a20 con 0x7f9b740a5fb0 2023-09-30T00:52:30.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.852+0000 7f9b79e77700 1 --2- 172.21.15.40:0/4054284686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b740a5fb0 0x7f9b7413e740 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9b7000c5d0 tx=0x7f9b70007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:30.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.852+0000 7f9b69ffb700 10 client.?.objecter ms_handle_connect 0x7f9b740a5fb0 2023-09-30T00:52:30.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.852+0000 7f9b69ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:30.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b69ffb700 1 -- 172.21.15.40:0/4054284686 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b70011590 con 0x7f9b740a5fb0 2023-09-30T00:52:30.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:30.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b69ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:52:30.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b69ffb700 10 monclient: dump: 2023-09-30T00:52:30.858 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:30.858 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:30.858 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:30.858 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:30.858 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:30.858 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:30.858 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:30.858 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:30.858 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:30.858 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:30.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b69ffb700 10 monclient: _finish_auth 0 2023-09-30T00:52:30.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b69ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:30.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b69ffb700 1 -- 172.21.15.40:0/4054284686 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b70011710 con 0x7f9b740a5fb0 2023-09-30T00:52:30.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b69ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:30.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b69ffb700 1 -- 172.21.15.40:0/4054284686 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b70011ad0 con 0x7f9b740a5fb0 2023-09-30T00:52:30.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b7c0db700 5 monclient: authenticate success, global_id 4734 2023-09-30T00:52:30.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b7c0db700 10 monclient: _renew_subs 2023-09-30T00:52:30.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b7c0db700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:30.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b7c0db700 1 -- 172.21.15.40:0/4054284686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b740a8b60 con 0x7f9b740a5fb0 2023-09-30T00:52:30.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:30.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b6bfff700 4 set_mon_vals no callback set 2023-09-30T00:52:30.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b69ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:52:30.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.856+0000 7f9b69ffb700 10 monclient: dump: 2023-09-30T00:52:30.860 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:30.860 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:30.860 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:30.860 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:30.860 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:30.860 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:30.860 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:30.860 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:30.860 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:30.860 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:30.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.857+0000 7f9b7c0db700 10 client.4734.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:30.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.857+0000 7f9b7c0db700 10 monclient: _renew_subs 2023-09-30T00:52:30.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.857+0000 7f9b7c0db700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:30.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.857+0000 7f9b7c0db700 1 -- 172.21.15.40:0/4054284686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b740aa1c0 con 0x7f9b740a5fb0 2023-09-30T00:52:30.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.857+0000 7f9b7c0db700 1 librados: init done 2023-09-30T00:52:30.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.858+0000 7f9b7c0db700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:30.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.858+0000 7f9b7c0db700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:30.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.858+0000 7f9b7c0db700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:30.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.858+0000 7f9b7c0db700 1 -- 172.21.15.40:0/4054284686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b740a9830 con 0x7f9b740a5fb0 2023-09-30T00:52:30.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.895+0000 7f9b69ffb700 1 -- 172.21.15.40:0/4054284686 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9b7001c460 con 0x7f9b740a5fb0 2023-09-30T00:52:30.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.895+0000 7f9b69ffb700 1 --2- 172.21.15.40:0/4054284686 >> v2:172.21.15.135:6800/116426 conn(0x7f9b580689e0 0x7f9b5806ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:30.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.896+0000 7f9b69ffb700 1 -- 172.21.15.40:0/4054284686 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9b700845b0 con 0x7f9b740a5fb0 2023-09-30T00:52:30.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.896+0000 7f9b69ffb700 10 client.4734.objecter ms_dispatch 0x7f9b74001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:30.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.896+0000 7f9b69ffb700 3 client.4734.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:30.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.896+0000 7f9b69ffb700 3 client.4734.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:30.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.896+0000 7f9b69ffb700 20 client.4734.objecter dump_active .. 0 homeless 2023-09-30T00:52:30.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.896+0000 7f9b79676700 1 --2- 172.21.15.40:0/4054284686 >> v2:172.21.15.135:6800/116426 conn(0x7f9b580689e0 0x7f9b5806ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:30.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.896+0000 7f9b79676700 10 monclient: get_auth_request con 0x7f9b580689e0 auth_method 0 2023-09-30T00:52:30.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.896+0000 7f9b79676700 1 --2- 172.21.15.40:0/4054284686 >> v2:172.21.15.135:6800/116426 conn(0x7f9b580689e0 0x7f9b5806ae90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9b740a8ee0 tx=0x7f9b64009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:30.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:30.896+0000 7f9b69ffb700 10 client.4734.objecter ms_handle_connect 0x7f9b580689e0 2023-09-30T00:52:31.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.154+0000 7f9b69ffb700 1 -- 172.21.15.40:0/4054284686 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9b70051ed0 con 0x7f9b740a5fb0 2023-09-30T00:52:31.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.154+0000 7f9b69ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:31.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.154+0000 7f9b69ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:31.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.413+0000 7f9b7c0db700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:31.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.413+0000 7f9b7c0db700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:31.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.413+0000 7f9b7c0db700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:31.414 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.413+0000 7f9b7c0db700 1 -- 172.21.15.40:0/4054284686 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9b74064a00 con 0x7f9b740a5fb0 2023-09-30T00:52:31.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.421+0000 7f9b69ffb700 1 -- 172.21.15.40:0/4054284686 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9b700554f0 con 0x7f9b740a5fb0 2023-09-30T00:52:31.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.421+0000 7f9b69ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:31.422 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.421+0000 7f9b69ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:31.422 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:31.422 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:31.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.424+0000 7f9b7c0db700 10 librados: watch_flush enter 2023-09-30T00:52:31.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.424+0000 7f9b7c0db700 10 librados: watch_flush exit 2023-09-30T00:52:31.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.424+0000 7f9b7c0db700 20 client.4734.objecter shutdown clearing up homeless session... 2023-09-30T00:52:31.424 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.424+0000 7f9b7c0db700 10 client.4734.objecter successfully canceled tick 2023-09-30T00:52:31.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.424+0000 7f9b7c0db700 1 -- 172.21.15.40:0/4054284686 >> v2:172.21.15.135:6800/116426 conn(0x7f9b580689e0 msgr2=0x7f9b5806ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:31.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.424+0000 7f9b7c0db700 1 --2- 172.21.15.40:0/4054284686 >> v2:172.21.15.135:6800/116426 conn(0x7f9b580689e0 0x7f9b5806ae90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9b740a8ee0 tx=0x7f9b64009040 comp rx=0 tx=0).stop 2023-09-30T00:52:31.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.424+0000 7f9b7c0db700 10 monclient: shutdown 2023-09-30T00:52:31.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.424+0000 7f9b7c0db700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:31.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.424+0000 7f9b7c0db700 1 -- 172.21.15.40:0/4054284686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b740a5fb0 msgr2=0x7f9b7413e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:31.425 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.424+0000 7f9b7c0db700 1 --2- 172.21.15.40:0/4054284686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b740a5fb0 0x7f9b7413e740 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9b7000c5d0 tx=0x7f9b70007f60 comp rx=0 tx=0).stop 2023-09-30T00:52:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.425+0000 7f9b7c0db700 1 -- 172.21.15.40:0/4054284686 shutdown_connections 2023-09-30T00:52:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.425+0000 7f9b7c0db700 1 --2- 172.21.15.40:0/4054284686 >> v2:172.21.15.135:6800/116426 conn(0x7f9b580689e0 0x7f9b5806ae90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.425+0000 7f9b7c0db700 1 --2- 172.21.15.40:0/4054284686 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9b740b1830 0x7f9b740a99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.425+0000 7f9b7c0db700 1 --2- 172.21.15.40:0/4054284686 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9b740a5fb0 0x7f9b7413e740 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.425+0000 7f9b7c0db700 1 --2- 172.21.15.40:0/4054284686 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9b740b1460 0x7f9b74151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.425+0000 7f9b7c0db700 1 -- 172.21.15.40:0/4054284686 >> 172.21.15.40:0/4054284686 conn(0x7f9b740631f0 msgr2=0x7f9b740b34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.425+0000 7f9b7c0db700 1 -- 172.21.15.40:0/4054284686 shutdown_connections 2023-09-30T00:52:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.425+0000 7f9b7c0db700 1 -- 172.21.15.40:0/4054284686 wait complete. 2023-09-30T00:52:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:31.425+0000 7f9b7c0db700 1 librados: shutdown 2023-09-30T00:52:34.048 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:06:09.759 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:34.051 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:06:09.763 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:34.081 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:06:09.792 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:34.106 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:06:09.792 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:34.108 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:06:09.795 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:34.110 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:06:09.822 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:34.113 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:06:09.825 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:34.132 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:06:09.819 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:34.138 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:06:09.850 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:34.440 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:34.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.575+0000 7fbe9ca88700 10 monclient: get_monmap_and_config 2023-09-30T00:52:34.576 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.575+0000 7fbe9ca88700 10 monclient: build_initial_monmap 2023-09-30T00:52:34.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.575+0000 7fbe9ca88700 10 monclient: monmap: 2023-09-30T00:52:34.577 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:34.577 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:34.577 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:34.576964+0000 2023-09-30T00:52:34.577 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:34.576964+0000 2023-09-30T00:52:34.577 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:34.578 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:34.578 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:34.578 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:34.578 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:34.578 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:34.578 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:34.578 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:34.581 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.576+0000 7fbe9ca88700 1 Processor -- start 2023-09-30T00:52:34.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.576+0000 7fbe9ca88700 1 -- start start 2023-09-30T00:52:34.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.576+0000 7fbe9ca88700 10 monclient: init 2023-09-30T00:52:34.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.577+0000 7fbe9ca88700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:34.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.577+0000 7fbe9ca88700 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-09-30T00:52:34.582 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.577+0000 7fbe9ca88700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbe980b1460 0x7fbe980b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:34.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.577+0000 7fbe9ca88700 10 monclient(hunting): picked mon.noname-c con 0x7fbe980b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:52:34.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.577+0000 7fbe9ca88700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe980a5fb0 addr v1:172.21.15.40:6789/0 2023-09-30T00:52:34.583 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.577+0000 7fbe9ca88700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9813ac20 0x7fbe9813b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:34.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.577+0000 7fbe9ca88700 10 monclient(hunting): picked mon.noname-b con 0x7fbe9813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:34.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.577+0000 7fbe9ca88700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe980a5b20 con 0x7fbe980a5fb0 2023-09-30T00:52:34.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.577+0000 7fbe9ca88700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:34.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.577+0000 7fbe9ca88700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe980a71f0 con 0x7fbe980b1460 2023-09-30T00:52:34.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.577+0000 7fbe9ca88700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:34.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.577+0000 7fbe9ca88700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980a74d0 con 0x7fbe9813ac20 2023-09-30T00:52:34.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.577+0000 7fbe9ca88700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:34.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.577+0000 7fbe9ca88700 10 monclient(hunting): authenticate will time out at 5615.878491s 2023-09-30T00:52:34.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.578+0000 7fbe96d9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9813ac20 0x7fbe9813b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:34.584 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.578+0000 7fbe9659c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fbe980b1460 0x7fbe980b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.578+0000 7fbe95d9b700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbe980a5fb0 0x7fbe98141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53008/0 (socket says 172.21.15.40:53008) 2023-09-30T00:52:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.578+0000 7fbe95d9b700 1 -- 172.21.15.40:0/1838609562 learned_addr learned my addr 172.21.15.40:0/1838609562 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:52:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.578+0000 7fbe96d9d700 10 monclient(hunting): get_auth_request con 0x7fbe9813ac20 auth_method 0 2023-09-30T00:52:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.578+0000 7fbe96d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.578+0000 7fbe96d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.578+0000 7fbe96d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.578+0000 7fbe9659c700 10 monclient(hunting): get_auth_request con 0x7fbe980b1460 auth_method 0 2023-09-30T00:52:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.578+0000 7fbe9659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:34.585 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.578+0000 7fbe9659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.578+0000 7fbe9659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.579+0000 7fbe9659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.579+0000 7fbe9659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.579+0000 7fbe9659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.579+0000 7fbe96d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.579+0000 7fbe96d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.579+0000 7fbe96d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:34.586 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.581+0000 7fbe9559a700 1 -- 172.21.15.40:0/1838609562 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755866417 0 0) 0x7fbe980a5b20 con 0x7fbe980a5fb0 2023-09-30T00:52:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.581+0000 7fbe9559a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.581+0000 7fbe9559a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.581+0000 7fbe9559a700 10 monclient(hunting): my global_id is 4737 2023-09-30T00:52:34.587 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.581+0000 7fbe9559a700 1 -- 172.21.15.40:0/1838609562 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe84003570 con 0x7fbe980a5fb0 2023-09-30T00:52:34.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.581+0000 7fbe9659c700 10 monclient(hunting): handle_auth_done global_id 4813 payload 274 2023-09-30T00:52:34.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.581+0000 7fbe9659c700 1 -- 172.21.15.40:0/1838609562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9813ac20 msgr2=0x7fbe9813b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:52:34.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.581+0000 7fbe9659c700 1 --2- 172.21.15.40:0/1838609562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9813ac20 0x7fbe9813b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:34.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.581+0000 7fbe9659c700 1 -- 172.21.15.40:0/1838609562 >> v1:172.21.15.40:6789/0 conn(0x7fbe980a5fb0 legacy=0x7fbe98141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:34.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.581+0000 7fbe9659c700 10 monclient: _finish_hunting 0 2023-09-30T00:52:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.581+0000 7fbe9659c700 1 monclient: found mon.noname-c 2023-09-30T00:52:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.581+0000 7fbe9659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.581+0000 7fbe9659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:52:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.581+0000 7fbe9659c700 1 -- 172.21.15.40:0/1838609562 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe980a63c0 con 0x7fbe980b1460 2023-09-30T00:52:34.589 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.584+0000 7fbe9659c700 1 --2- 172.21.15.40:0/1838609562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe980b1460 0x7fbe980b1830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbe800098e0 tx=0x7fbe800091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa76e8269f2d8aac server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9559a700 1 -- 172.21.15.40:0/1838609562 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe80011630 con 0x7fbe980b1460 2023-09-30T00:52:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:34.594 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:52:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9559a700 10 monclient: dump: 2023-09-30T00:52:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:34.595 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:34.595 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:34.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9559a700 10 monclient: _finish_auth 0 2023-09-30T00:52:34.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:34.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9559a700 1 -- 172.21.15.40:0/1838609562 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe80011790 con 0x7fbe980b1460 2023-09-30T00:52:34.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:34.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9559a700 1 -- 172.21.15.40:0/1838609562 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe80011b30 con 0x7fbe980b1460 2023-09-30T00:52:34.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:52:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9559a700 10 monclient: dump: 2023-09-30T00:52:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:34.598 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:34.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9ca88700 5 monclient: authenticate success, global_id 4813 2023-09-30T00:52:34.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9ca88700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:34.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.588+0000 7fbe9ca88700 4 set_mon_vals no callback set 2023-09-30T00:52:34.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.589+0000 7fbe9ca88700 10 monclient: shutdown 2023-09-30T00:52:34.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.589+0000 7fbe9ca88700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:34.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.589+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1838609562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe980b1460 msgr2=0x7fbe980b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:34.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.589+0000 7fbe9ca88700 1 --2- 172.21.15.40:0/1838609562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe980b1460 0x7fbe980b1830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbe800098e0 tx=0x7fbe800091a0 comp rx=0 tx=0).stop 2023-09-30T00:52:34.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.589+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1838609562 shutdown_connections 2023-09-30T00:52:34.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.589+0000 7fbe9ca88700 1 --2- 172.21.15.40:0/1838609562 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe9813ac20 0x7fbe9813b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:34.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.589+0000 7fbe9ca88700 1 --2- 172.21.15.40:0/1838609562 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe980b1460 0x7fbe980b1830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:34.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.589+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1838609562 >> 172.21.15.40:0/1838609562 conn(0x7fbe980631f0 msgr2=0x7fbe980635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:34.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.589+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1838609562 shutdown_connections 2023-09-30T00:52:34.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.589+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1838609562 wait complete. 2023-09-30T00:52:34.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.590+0000 7fbe9ca88700 10 monclient: build_initial_monmap 2023-09-30T00:52:34.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.590+0000 7fbe9ca88700 10 monclient: monmap: 2023-09-30T00:52:34.600 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:34.600 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:34.600 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:34.600 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:34.600 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:34.600 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:34.600 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:34.600 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:34.600 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:34.601 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:34.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.590+0000 7fbe9ca88700 1 librados: starting msgr at 2023-09-30T00:52:34.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.590+0000 7fbe9ca88700 1 librados: starting objecter 2023-09-30T00:52:34.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.590+0000 7fbe9ca88700 1 Processor -- start 2023-09-30T00:52:34.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.590+0000 7fbe9ca88700 1 -- start start 2023-09-30T00:52:34.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.590+0000 7fbe9ca88700 1 librados: setting wanted keys 2023-09-30T00:52:34.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.590+0000 7fbe9ca88700 1 librados: calling monclient init 2023-09-30T00:52:34.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.590+0000 7fbe9ca88700 10 monclient: init 2023-09-30T00:52:34.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:34.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:52:34.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe980a5fb0 0x7fbe980a9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:34.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 10 monclient(hunting): picked mon.noname-a con 0x7fbe980a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:34.601 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe980b1460 0x7fbe9813e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:34.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 10 monclient(hunting): picked mon.noname-c con 0x7fbe980b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:34.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe9813ac20 0x7fbe980aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:34.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 10 monclient(hunting): picked mon.noname-b con 0x7fbe9813ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:34.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:34.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980abac0 con 0x7fbe980a5fb0 2023-09-30T00:52:34.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:34.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980abcc0 con 0x7fbe9813ac20 2023-09-30T00:52:34.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:34.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980a9c60 con 0x7fbe980b1460 2023-09-30T00:52:34.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:34.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.591+0000 7fbe9ca88700 10 monclient(hunting): authenticate will time out at 5615.892541s 2023-09-30T00:52:34.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe95d9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe980b1460 0x7fbe9813e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:34.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe9659c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe980a5fb0 0x7fbe980a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:34.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe96d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe9813ac20 0x7fbe980aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:34.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe95d9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe980b1460 0x7fbe9813e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49020/0 (socket says 172.21.15.40:49020) 2023-09-30T00:52:34.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe95d9b700 1 -- 172.21.15.40:0/1948636141 learned_addr learned my addr 172.21.15.40:0/1948636141 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:34.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe95d9b700 10 monclient(hunting): get_auth_request con 0x7fbe980b1460 auth_method 0 2023-09-30T00:52:34.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe95d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:34.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe95d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:34.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe95d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:34.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe96d9d700 10 monclient(hunting): get_auth_request con 0x7fbe9813ac20 auth_method 0 2023-09-30T00:52:34.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe96d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:34.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe96d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:34.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe96d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:34.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe9659c700 10 monclient(hunting): get_auth_request con 0x7fbe980a5fb0 auth_method 0 2023-09-30T00:52:34.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe9659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:34.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe9659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:34.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.592+0000 7fbe9659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:34.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.593+0000 7fbe95d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:34.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.593+0000 7fbe95d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:34.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.593+0000 7fbe95d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:34.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.593+0000 7fbe96d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:34.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.593+0000 7fbe96d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:34.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.593+0000 7fbe96d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:34.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.593+0000 7fbe9659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:34.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.593+0000 7fbe9659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:34.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.593+0000 7fbe9659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:34.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.596+0000 7fbe96d9d700 10 monclient(hunting): handle_auth_done global_id 4816 payload 824 2023-09-30T00:52:34.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.596+0000 7fbe96d9d700 1 -- 172.21.15.40:0/1948636141 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe980b1460 msgr2=0x7fbe9813e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:34.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.596+0000 7fbe96d9d700 1 --2- 172.21.15.40:0/1948636141 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe980b1460 0x7fbe9813e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.596+0000 7fbe96d9d700 1 -- 172.21.15.40:0/1948636141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe980a5fb0 msgr2=0x7fbe980a9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.596+0000 7fbe96d9d700 1 --2- 172.21.15.40:0/1948636141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe980a5fb0 0x7fbe980a9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.596+0000 7fbe96d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:52:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.596+0000 7fbe96d9d700 1 monclient: found mon.noname-b 2023-09-30T00:52:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.596+0000 7fbe96d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.596+0000 7fbe96d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:52:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.596+0000 7fbe96d9d700 1 -- 172.21.15.40:0/1948636141 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe80009420 con 0x7fbe9813ac20 2023-09-30T00:52:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.598+0000 7fbe96d9d700 1 --2- 172.21.15.40:0/1948636141 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe9813ac20 0x7fbe980aa1d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbe9000ecf0 tx=0x7fbe9000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.598+0000 7fbe8e7fc700 10 client.?.objecter ms_handle_connect 0x7fbe9813ac20 2023-09-30T00:52:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.598+0000 7fbe8e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe8e7fc700 1 -- 172.21.15.40:0/1948636141 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9000c890 con 0x7fbe9813ac20 2023-09-30T00:52:34.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe8e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:34.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe8e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:52:34.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe8e7fc700 10 monclient: dump: 2023-09-30T00:52:34.608 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:34.608 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:34.608 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:34.608 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:34.608 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:34.608 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:34.608 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:34.608 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:34.609 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:34.609 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:34.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe8e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:52:34.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe8e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:34.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe8e7fc700 1 -- 172.21.15.40:0/1948636141 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9000f040 con 0x7fbe9813ac20 2023-09-30T00:52:34.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe9ca88700 5 monclient: authenticate success, global_id 4816 2023-09-30T00:52:34.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe9ca88700 10 monclient: _renew_subs 2023-09-30T00:52:34.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe9ca88700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:34.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1948636141 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe980a8a40 con 0x7fbe9813ac20 2023-09-30T00:52:34.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe8e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:34.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe8e7fc700 1 -- 172.21.15.40:0/1948636141 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9000cc60 con 0x7fbe9813ac20 2023-09-30T00:52:34.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe8e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe8e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:52:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe8e7fc700 10 monclient: dump: 2023-09-30T00:52:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:34.611 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe94d99700 4 set_mon_vals no callback set 2023-09-30T00:52:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe9ca88700 10 client.4816.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe9ca88700 10 monclient: _renew_subs 2023-09-30T00:52:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe9ca88700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1948636141 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe980aa710 con 0x7fbe9813ac20 2023-09-30T00:52:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.602+0000 7fbe9ca88700 1 librados: init done 2023-09-30T00:52:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.604+0000 7fbe9ca88700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.604+0000 7fbe9ca88700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.604+0000 7fbe9ca88700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:34.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.604+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1948636141 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe980a9e20 con 0x7fbe9813ac20 2023-09-30T00:52:34.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.624+0000 7fbe8e7fc700 1 -- 172.21.15.40:0/1948636141 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fbe90014480 con 0x7fbe9813ac20 2023-09-30T00:52:34.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.624+0000 7fbe8e7fc700 1 --2- 172.21.15.40:0/1948636141 >> v2:172.21.15.135:6800/116426 conn(0x7fbe78068960 0x7fbe7806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:34.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.624+0000 7fbe9659c700 1 --2- 172.21.15.40:0/1948636141 >> v2:172.21.15.135:6800/116426 conn(0x7fbe78068960 0x7fbe7806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:34.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.624+0000 7fbe9659c700 10 monclient: get_auth_request con 0x7fbe78068960 auth_method 0 2023-09-30T00:52:34.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.625+0000 7fbe9659c700 1 --2- 172.21.15.40:0/1948636141 >> v2:172.21.15.135:6800/116426 conn(0x7fbe78068960 0x7fbe7806ae10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbe800093f0 tx=0x7fbe80017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:34.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.625+0000 7fbe8e7fc700 10 client.4816.objecter ms_handle_connect 0x7fbe78068960 2023-09-30T00:52:34.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.794+0000 7fbe8e7fc700 1 -- 172.21.15.40:0/1948636141 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fbe90056b70 con 0x7fbe9813ac20 2023-09-30T00:52:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.794+0000 7fbe8e7fc700 10 client.4816.objecter ms_dispatch 0x7fbe98001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.794+0000 7fbe8e7fc700 3 client.4816.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.794+0000 7fbe8e7fc700 3 client.4816.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:34.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.794+0000 7fbe8e7fc700 20 client.4816.objecter dump_active .. 0 homeless 2023-09-30T00:52:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.877+0000 7fbe8e7fc700 1 -- 172.21.15.40:0/1948636141 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbe900525e0 con 0x7fbe9813ac20 2023-09-30T00:52:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.877+0000 7fbe8e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:34.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:34.877+0000 7fbe8e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:35.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.151+0000 7fbe9ca88700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:35.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.151+0000 7fbe9ca88700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:35.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.151+0000 7fbe9ca88700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:35.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.151+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1948636141 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbe98064a00 con 0x7fbe9813ac20 2023-09-30T00:52:35.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.157+0000 7fbe8e7fc700 1 -- 172.21.15.40:0/1948636141 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbe90018080 con 0x7fbe9813ac20 2023-09-30T00:52:35.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.157+0000 7fbe8e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:35.157 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.157+0000 7fbe8e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:35.157 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:35.158 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:35.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 10 librados: watch_flush enter 2023-09-30T00:52:35.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 10 librados: watch_flush exit 2023-09-30T00:52:35.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 20 client.4816.objecter shutdown clearing up homeless session... 2023-09-30T00:52:35.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 10 client.4816.objecter successfully canceled tick 2023-09-30T00:52:35.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1948636141 >> v2:172.21.15.135:6800/116426 conn(0x7fbe78068960 msgr2=0x7fbe7806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:35.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 1 --2- 172.21.15.40:0/1948636141 >> v2:172.21.15.135:6800/116426 conn(0x7fbe78068960 0x7fbe7806ae10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbe800093f0 tx=0x7fbe80017000 comp rx=0 tx=0).stop 2023-09-30T00:52:35.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 10 monclient: shutdown 2023-09-30T00:52:35.160 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:35.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1948636141 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe9813ac20 msgr2=0x7fbe980aa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:35.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 1 --2- 172.21.15.40:0/1948636141 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe9813ac20 0x7fbe980aa1d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbe9000ecf0 tx=0x7fbe9000b040 comp rx=0 tx=0).stop 2023-09-30T00:52:35.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1948636141 shutdown_connections 2023-09-30T00:52:35.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 1 --2- 172.21.15.40:0/1948636141 >> v2:172.21.15.135:6800/116426 conn(0x7fbe78068960 0x7fbe7806ae10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:35.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 1 --2- 172.21.15.40:0/1948636141 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fbe9813ac20 0x7fbe980aa1d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:35.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 1 --2- 172.21.15.40:0/1948636141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe980a5fb0 0x7fbe980a9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:35.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 1 --2- 172.21.15.40:0/1948636141 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fbe980b1460 0x7fbe9813e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:35.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1948636141 >> 172.21.15.40:0/1948636141 conn(0x7fbe980631f0 msgr2=0x7fbe980b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:35.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1948636141 shutdown_connections 2023-09-30T00:52:35.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 1 -- 172.21.15.40:0/1948636141 wait complete. 2023-09-30T00:52:35.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:35.159+0000 7fbe9ca88700 1 librados: shutdown 2023-09-30T00:52:38.175 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:38.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.324+0000 7fd6e3f98700 10 monclient: get_monmap_and_config 2023-09-30T00:52:38.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.324+0000 7fd6e3f98700 10 monclient: build_initial_monmap 2023-09-30T00:52:38.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.324+0000 7fd6e3f98700 10 monclient: monmap: 2023-09-30T00:52:38.325 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:38.325 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:38.325 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:38.325597+0000 2023-09-30T00:52:38.325 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:38.325597+0000 2023-09-30T00:52:38.325 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:38.325 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:38.326 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:38.326 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:38.326 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:38.326 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:38.326 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:38.326 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:38.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.325+0000 7fd6e3f98700 1 Processor -- start 2023-09-30T00:52:38.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.325+0000 7fd6e3f98700 1 -- start start 2023-09-30T00:52:38.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.325+0000 7fd6e3f98700 10 monclient: init 2023-09-30T00:52:38.327 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.326+0000 7fd6e3f98700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:38.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.326+0000 7fd6e3f98700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-09-30T00:52:38.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.326+0000 7fd6e3f98700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6dc0b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:52:38.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.326+0000 7fd6e3f98700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6dc0b1830 addr v1:172.21.15.135:6789/0 2023-09-30T00:52:38.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.326+0000 7fd6e3f98700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fd6dc0a5fb0 0x7fd6dc13b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:38.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.326+0000 7fd6e3f98700 10 monclient(hunting): picked mon.noname-c con 0x7fd6dc0a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:52:38.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.326+0000 7fd6e3f98700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6dc0a5b20 con 0x7fd6dc0b1460 2023-09-30T00:52:38.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.326+0000 7fd6e3f98700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6dc0a71f0 con 0x7fd6dc0b1830 2023-09-30T00:52:38.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.326+0000 7fd6e3f98700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:38.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.326+0000 7fd6e3f98700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6dc141e40 con 0x7fd6dc0a5fb0 2023-09-30T00:52:38.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.326+0000 7fd6e3f98700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:38.328 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.326+0000 7fd6e3f98700 10 monclient(hunting): authenticate will time out at 5619.627753s 2023-09-30T00:52:38.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.349+0000 7fd6e1d34700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd6dc0b1460 0x7fd6dc141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53014/0 (socket says 172.21.15.40:53014) 2023-09-30T00:52:38.349 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.349+0000 7fd6e1d34700 1 -- 172.21.15.40:0/8045404 learned_addr learned my addr 172.21.15.40:0/8045404 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:52:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.356+0000 7fd6e2535700 1 --2- 172.21.15.40:0/8045404 >> v2:172.21.15.135:3300/0 conn(0x7fd6dc0a5fb0 0x7fd6dc13b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.357+0000 7fd6e2535700 10 monclient(hunting): get_auth_request con 0x7fd6dc0a5fb0 auth_method 0 2023-09-30T00:52:38.357 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.357+0000 7fd6e2535700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:38.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.357+0000 7fd6e2535700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:38.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.357+0000 7fd6e2535700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:38.358 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.358+0000 7fd6e2535700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:38.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.358+0000 7fd6e2535700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:38.359 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.358+0000 7fd6e2535700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:38.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.363+0000 7fd6e2535700 10 monclient(hunting): handle_auth_done global_id 4819 payload 274 2023-09-30T00:52:38.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.363+0000 7fd6e2535700 1 -- 172.21.15.40:0/8045404 >> v1:172.21.15.135:6789/0 conn(0x7fd6dc0b1830 legacy=0x7fd6dc13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:38.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.363+0000 7fd6e2535700 1 -- 172.21.15.40:0/8045404 >> v1:172.21.15.40:6789/0 conn(0x7fd6dc0b1460 legacy=0x7fd6dc141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:52:38.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.363+0000 7fd6e2535700 10 monclient: _finish_hunting 0 2023-09-30T00:52:38.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.363+0000 7fd6e2535700 1 monclient: found mon.noname-c 2023-09-30T00:52:38.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.363+0000 7fd6e2535700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.363+0000 7fd6e2535700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:52:38.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.363+0000 7fd6e2535700 1 -- 172.21.15.40:0/8045404 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc0a74d0 con 0x7fd6dc0a5fb0 2023-09-30T00:52:38.366 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.366+0000 7fd6e2535700 1 --2- 172.21.15.40:0/8045404 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6dc0a5fb0 0x7fd6dc13b330 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009e90 tx=0x7fd6d4009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca7c27dc104c79c8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.371+0000 7fd6e0d32700 1 -- 172.21.15.40:0/8045404 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d400e070 con 0x7fd6dc0a5fb0 2023-09-30T00:52:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.371+0000 7fd6e0d32700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.371+0000 7fd6e0d32700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:52:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e0d32700 10 monclient: dump: 2023-09-30T00:52:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:38.373 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:38.374 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e0d32700 10 monclient: _finish_auth 0 2023-09-30T00:52:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e0d32700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e0d32700 1 -- 172.21.15.40:0/8045404 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d400bbb0 con 0x7fd6dc0a5fb0 2023-09-30T00:52:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e3f98700 5 monclient: authenticate success, global_id 4819 2023-09-30T00:52:38.374 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e3f98700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:52:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e0d32700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e3f98700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e0d32700 1 -- 172.21.15.40:0/8045404 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4012430 con 0x7fd6dc0a5fb0 2023-09-30T00:52:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e3f98700 4 set_mon_vals no callback set 2023-09-30T00:52:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e0d32700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e0d32700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:52:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e0d32700 10 monclient: dump: 2023-09-30T00:52:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:38.375 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:38.376 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:38.376 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:38.376 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:38.376 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:38.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e3f98700 10 monclient: shutdown 2023-09-30T00:52:38.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e3f98700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:38.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e3f98700 1 -- 172.21.15.40:0/8045404 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6dc0a5fb0 msgr2=0x7fd6dc13b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:38.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e3f98700 1 --2- 172.21.15.40:0/8045404 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6dc0a5fb0 0x7fd6dc13b330 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009e90 tx=0x7fd6d4009780 comp rx=0 tx=0).stop 2023-09-30T00:52:38.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e3f98700 1 -- 172.21.15.40:0/8045404 shutdown_connections 2023-09-30T00:52:38.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e3f98700 1 --2- 172.21.15.40:0/8045404 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6dc0a5fb0 0x7fd6dc13b330 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:38.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.372+0000 7fd6e3f98700 1 -- 172.21.15.40:0/8045404 >> 172.21.15.40:0/8045404 conn(0x7fd6dc0631f0 msgr2=0x7fd6dc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:38.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.373+0000 7fd6e3f98700 1 -- 172.21.15.40:0/8045404 shutdown_connections 2023-09-30T00:52:38.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.373+0000 7fd6e3f98700 1 -- 172.21.15.40:0/8045404 wait complete. 2023-09-30T00:52:38.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.373+0000 7fd6e3f98700 10 monclient: build_initial_monmap 2023-09-30T00:52:38.379 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.373+0000 7fd6e3f98700 10 monclient: monmap: 2023-09-30T00:52:38.379 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:38.379 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:38.379 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:38.380 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:38.380 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:38.380 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:38.380 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:38.380 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:38.380 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:38.380 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:38.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.373+0000 7fd6e3f98700 1 librados: starting msgr at 2023-09-30T00:52:38.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.373+0000 7fd6e3f98700 1 librados: starting objecter 2023-09-30T00:52:38.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 1 Processor -- start 2023-09-30T00:52:38.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 1 -- start start 2023-09-30T00:52:38.380 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 1 librados: setting wanted keys 2023-09-30T00:52:38.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 1 librados: calling monclient init 2023-09-30T00:52:38.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 10 monclient: init 2023-09-30T00:52:38.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:38.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:52:38.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6dc0a5fb0 0x7fd6dc13e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:38.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 10 monclient(hunting): picked mon.noname-b con 0x7fd6dc0a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:38.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6dc0b1460 0x7fd6dc150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:38.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 10 monclient(hunting): picked mon.noname-c con 0x7fd6dc0b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:38.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6dc0b1830 0x7fd6dc139cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:38.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 10 monclient(hunting): picked mon.noname-a con 0x7fd6dc0b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:38.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:38.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc0ab880 con 0x7fd6dc0b1830 2023-09-30T00:52:38.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:38.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc151790 con 0x7fd6dc0a5fb0 2023-09-30T00:52:38.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:38.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc151c20 con 0x7fd6dc0b1460 2023-09-30T00:52:38.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:38.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.374+0000 7fd6e3f98700 10 monclient(hunting): authenticate will time out at 5619.675757s 2023-09-30T00:52:38.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.375+0000 7fd6e2535700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6dc0b1830 0x7fd6dc139cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:38.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.375+0000 7fd6e1533700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6dc0b1460 0x7fd6dc150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:38.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.375+0000 7fd6e1533700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6dc0b1460 0x7fd6dc150df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49028/0 (socket says 172.21.15.40:49028) 2023-09-30T00:52:38.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.375+0000 7fd6e1533700 1 -- 172.21.15.40:0/3372405248 learned_addr learned my addr 172.21.15.40:0/3372405248 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:38.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.375+0000 7fd6e1533700 10 monclient(hunting): get_auth_request con 0x7fd6dc0b1460 auth_method 0 2023-09-30T00:52:38.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.375+0000 7fd6e1533700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:38.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.375+0000 7fd6e1533700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:38.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.375+0000 7fd6e1533700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:38.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.375+0000 7fd6e2535700 10 monclient(hunting): get_auth_request con 0x7fd6dc0b1830 auth_method 0 2023-09-30T00:52:38.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.375+0000 7fd6e2535700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:38.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.375+0000 7fd6e2535700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:38.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.375+0000 7fd6e2535700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:38.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.376+0000 7fd6e1533700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:38.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.376+0000 7fd6e1533700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:38.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.376+0000 7fd6e1533700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:38.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.376+0000 7fd6e1d34700 1 --2- 172.21.15.40:0/3372405248 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6dc0a5fb0 0x7fd6dc13e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:38.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.377+0000 7fd6e1d34700 10 monclient(hunting): get_auth_request con 0x7fd6dc0a5fb0 auth_method 0 2023-09-30T00:52:38.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.377+0000 7fd6e1d34700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:38.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.377+0000 7fd6e1d34700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:38.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.377+0000 7fd6e1d34700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:38.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.377+0000 7fd6e2535700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:38.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.377+0000 7fd6e2535700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:38.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.377+0000 7fd6e2535700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:38.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.380+0000 7fd6e1d34700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:38.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.380+0000 7fd6e1d34700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:38.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.380+0000 7fd6e1d34700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:38.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.380+0000 7fd6e1533700 10 monclient(hunting): handle_auth_done global_id 4652 payload 824 2023-09-30T00:52:38.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.380+0000 7fd6e1533700 1 -- 172.21.15.40:0/3372405248 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6dc0a5fb0 msgr2=0x7fd6dc13e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:38.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.380+0000 7fd6e1533700 1 --2- 172.21.15.40:0/3372405248 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6dc0a5fb0 0x7fd6dc13e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:38.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.380+0000 7fd6e1533700 1 -- 172.21.15.40:0/3372405248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6dc0b1830 msgr2=0x7fd6dc139cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:38.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.380+0000 7fd6e1533700 1 --2- 172.21.15.40:0/3372405248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6dc0b1830 0x7fd6dc139cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:38.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.380+0000 7fd6e1533700 10 monclient: _finish_hunting 0 2023-09-30T00:52:38.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.380+0000 7fd6e1533700 1 monclient: found mon.noname-c 2023-09-30T00:52:38.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.380+0000 7fd6e1533700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:38.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.380+0000 7fd6e1533700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:52:38.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.380+0000 7fd6e1533700 1 -- 172.21.15.40:0/3372405248 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d4009a20 con 0x7fd6dc0b1460 2023-09-30T00:52:38.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.384+0000 7fd6e1533700 1 --2- 172.21.15.40:0/3372405248 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6dc0b1460 0x7fd6dc150df0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00c4e0 tx=0x7fd6cc00c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:38.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.384+0000 7fd6d1ffb700 10 client.?.objecter ms_handle_connect 0x7fd6dc0b1460 2023-09-30T00:52:38.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.384+0000 7fd6d1ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:38.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6d1ffb700 1 -- 172.21.15.40:0/3372405248 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc011590 con 0x7fd6dc0b1460 2023-09-30T00:52:38.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:38.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6d1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:52:38.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6d1ffb700 10 monclient: dump: 2023-09-30T00:52:38.390 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:38.390 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:38.391 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:38.391 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:38.391 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:38.391 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:38.391 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:38.391 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:38.391 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:38.391 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:38.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6d1ffb700 10 monclient: _finish_auth 0 2023-09-30T00:52:38.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6d1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:38.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6d1ffb700 1 -- 172.21.15.40:0/3372405248 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6cc011710 con 0x7fd6dc0b1460 2023-09-30T00:52:38.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6d1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:38.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6d1ffb700 1 -- 172.21.15.40:0/3372405248 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc011ad0 con 0x7fd6dc0b1460 2023-09-30T00:52:38.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6e3f98700 5 monclient: authenticate success, global_id 4652 2023-09-30T00:52:38.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6e3f98700 10 monclient: _renew_subs 2023-09-30T00:52:38.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6e3f98700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:38.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6e3f98700 1 -- 172.21.15.40:0/3372405248 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6dc0aba80 con 0x7fd6dc0b1460 2023-09-30T00:52:38.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6d3fff700 4 set_mon_vals no callback set 2023-09-30T00:52:38.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:38.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6d1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:52:38.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.389+0000 7fd6d1ffb700 10 monclient: dump: 2023-09-30T00:52:38.392 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:38.392 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:38.392 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:38.392 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:38.393 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.390+0000 7fd6e3f98700 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.390+0000 7fd6e3f98700 10 monclient: _renew_subs 2023-09-30T00:52:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.390+0000 7fd6e3f98700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.390+0000 7fd6e3f98700 1 -- 172.21.15.40:0/3372405248 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6dc13a1f0 con 0x7fd6dc0b1460 2023-09-30T00:52:38.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.390+0000 7fd6e3f98700 1 librados: init done 2023-09-30T00:52:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.391+0000 7fd6e3f98700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.391+0000 7fd6e3f98700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.391+0000 7fd6e3f98700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.391+0000 7fd6e3f98700 1 -- 172.21.15.40:0/3372405248 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6dc151570 con 0x7fd6dc0b1460 2023-09-30T00:52:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.618+0000 7fd6d1ffb700 1 -- 172.21.15.40:0/3372405248 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd6cc01b3b0 con 0x7fd6dc0b1460 2023-09-30T00:52:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.619+0000 7fd6d1ffb700 1 --2- 172.21.15.40:0/3372405248 >> v2:172.21.15.135:6800/116426 conn(0x7fd6c0068890 0x7fd6c006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:38.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.619+0000 7fd6e1d34700 1 --2- 172.21.15.40:0/3372405248 >> v2:172.21.15.135:6800/116426 conn(0x7fd6c0068890 0x7fd6c006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.619+0000 7fd6e1d34700 10 monclient: get_auth_request con 0x7fd6c0068890 auth_method 0 2023-09-30T00:52:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.619+0000 7fd6e1d34700 1 --2- 172.21.15.40:0/3372405248 >> v2:172.21.15.135:6800/116426 conn(0x7fd6c0068890 0x7fd6c006ad40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8007880 tx=0x7fd6d8007f90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:38.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.620+0000 7fd6d1ffb700 10 client.4652.objecter ms_handle_connect 0x7fd6c0068890 2023-09-30T00:52:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.629+0000 7fd6d1ffb700 1 -- 172.21.15.40:0/3372405248 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd6cc055330 con 0x7fd6dc0b1460 2023-09-30T00:52:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.629+0000 7fd6d1ffb700 10 client.4652.objecter ms_dispatch 0x7fd6dc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.629+0000 7fd6d1ffb700 3 client.4652.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.629+0000 7fd6d1ffb700 3 client.4652.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:38.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.629+0000 7fd6d1ffb700 20 client.4652.objecter dump_active .. 0 homeless 2023-09-30T00:52:38.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.694+0000 7fd6d1ffb700 1 -- 172.21.15.40:0/3372405248 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd6cc050f70 con 0x7fd6dc0b1460 2023-09-30T00:52:38.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.694+0000 7fd6d1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:38.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.694+0000 7fd6d1ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:38.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.958+0000 7fd6e3f98700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.958+0000 7fd6e3f98700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.958+0000 7fd6e3f98700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.958+0000 7fd6e3f98700 1 -- 172.21.15.40:0/3372405248 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd6dc064a00 con 0x7fd6dc0b1460 2023-09-30T00:52:38.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.965+0000 7fd6d1ffb700 1 -- 172.21.15.40:0/3372405248 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd6cc017080 con 0x7fd6dc0b1460 2023-09-30T00:52:38.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.965+0000 7fd6d1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:38.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.965+0000 7fd6d1ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:38.967 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:38.967 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:38.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 10 librados: watch_flush enter 2023-09-30T00:52:38.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 10 librados: watch_flush exit 2023-09-30T00:52:38.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 20 client.4652.objecter shutdown clearing up homeless session... 2023-09-30T00:52:38.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 10 client.4652.objecter successfully canceled tick 2023-09-30T00:52:38.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 1 -- 172.21.15.40:0/3372405248 >> v2:172.21.15.135:6800/116426 conn(0x7fd6c0068890 msgr2=0x7fd6c006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:38.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 1 --2- 172.21.15.40:0/3372405248 >> v2:172.21.15.135:6800/116426 conn(0x7fd6c0068890 0x7fd6c006ad40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8007880 tx=0x7fd6d8007f90 comp rx=0 tx=0).stop 2023-09-30T00:52:38.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 10 monclient: shutdown 2023-09-30T00:52:38.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:38.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 1 -- 172.21.15.40:0/3372405248 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6dc0b1460 msgr2=0x7fd6dc150df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:38.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 1 --2- 172.21.15.40:0/3372405248 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6dc0b1460 0x7fd6dc150df0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00c4e0 tx=0x7fd6cc00c7f0 comp rx=0 tx=0).stop 2023-09-30T00:52:38.970 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 1 -- 172.21.15.40:0/3372405248 shutdown_connections 2023-09-30T00:52:38.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 1 --2- 172.21.15.40:0/3372405248 >> v2:172.21.15.135:6800/116426 conn(0x7fd6c0068890 0x7fd6c006ad40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:38.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 1 --2- 172.21.15.40:0/3372405248 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fd6dc0a5fb0 0x7fd6dc13e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:38.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 1 --2- 172.21.15.40:0/3372405248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6dc0b1830 0x7fd6dc139cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:38.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 1 --2- 172.21.15.40:0/3372405248 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fd6dc0b1460 0x7fd6dc150df0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:38.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 1 -- 172.21.15.40:0/3372405248 >> 172.21.15.40:0/3372405248 conn(0x7fd6dc0631f0 msgr2=0x7fd6dc13b490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:38.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 1 -- 172.21.15.40:0/3372405248 shutdown_connections 2023-09-30T00:52:38.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.967+0000 7fd6e3f98700 1 -- 172.21.15.40:0/3372405248 wait complete. 2023-09-30T00:52:38.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:38.968+0000 7fd6e3f98700 1 librados: shutdown 2023-09-30T00:52:41.982 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.126+0000 7fa3389e3700 10 monclient: get_monmap_and_config 2023-09-30T00:52:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.126+0000 7fa3389e3700 10 monclient: build_initial_monmap 2023-09-30T00:52:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.126+0000 7fa3389e3700 10 monclient: monmap: 2023-09-30T00:52:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:42.127740+0000 2023-09-30T00:52:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:42.127740+0000 2023-09-30T00:52:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:42.128 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:42.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.127+0000 7fa3389e3700 1 Processor -- start 2023-09-30T00:52:42.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.127+0000 7fa3389e3700 1 -- start start 2023-09-30T00:52:42.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.127+0000 7fa3389e3700 10 monclient: init 2023-09-30T00:52:42.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa3389e3700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:42.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa3389e3700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-09-30T00:52:42.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa3389e3700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa3340b1460 0x7fa3340b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:42.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa3389e3700 10 monclient(hunting): picked mon.noname-a con 0x7fa3340b1460 addr v2:172.21.15.40:3300/0 2023-09-30T00:52:42.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa3389e3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3340a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:52:42.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa3389e3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa33413ac20 0x7fa33413b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:42.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa3389e3700 10 monclient(hunting): picked mon.noname-b con 0x7fa33413ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:42.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa3389e3700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3340a5b20 con 0x7fa3340a5fb0 2023-09-30T00:52:42.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa3389e3700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:42.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa3389e3700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3340a71f0 con 0x7fa3340b1460 2023-09-30T00:52:42.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa3389e3700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:42.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa3389e3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3340a74d0 con 0x7fa33413ac20 2023-09-30T00:52:42.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa3389e3700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:42.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa3389e3700 10 monclient(hunting): authenticate will time out at 5623.429357s 2023-09-30T00:52:42.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.128+0000 7fa33259c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa3340b1460 0x7fa3340b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:42.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.129+0000 7fa33259c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa3340b1460 0x7fa3340b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60790/0 (socket says 172.21.15.40:60790) 2023-09-30T00:52:42.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.129+0000 7fa33259c700 1 -- 172.21.15.40:0/600167138 learned_addr learned my addr 172.21.15.40:0/600167138 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:42.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.129+0000 7fa33259c700 10 monclient(hunting): get_auth_request con 0x7fa3340b1460 auth_method 0 2023-09-30T00:52:42.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.129+0000 7fa33259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:42.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.129+0000 7fa33259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:42.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.129+0000 7fa33259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:42.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.130+0000 7fa33259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:42.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.130+0000 7fa33259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:42.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.130+0000 7fa33259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:42.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.133+0000 7fa33259c700 10 monclient(hunting): handle_auth_done global_id 4746 payload 274 2023-09-30T00:52:42.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.133+0000 7fa33259c700 1 -- 172.21.15.40:0/600167138 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa33413ac20 msgr2=0x7fa33413b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:52:42.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.133+0000 7fa33259c700 1 --2- 172.21.15.40:0/600167138 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa33413ac20 0x7fa33413b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:42.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.133+0000 7fa33259c700 1 -- 172.21.15.40:0/600167138 >> v1:172.21.15.135:6789/0 conn(0x7fa3340a5fb0 legacy=0x7fa334141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:52:42.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.133+0000 7fa33259c700 10 monclient: _finish_hunting 0 2023-09-30T00:52:42.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.133+0000 7fa33259c700 1 monclient: found mon.noname-a 2023-09-30T00:52:42.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.133+0000 7fa33259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:42.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.133+0000 7fa33259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:52:42.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.133+0000 7fa33259c700 1 -- 172.21.15.40:0/600167138 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3340a63c0 con 0x7fa3340b1460 2023-09-30T00:52:42.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.135+0000 7fa33259c700 1 --2- 172.21.15.40:0/600167138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3340b1460 0x7fa3340b1830 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa324009f20 tx=0x7fa324009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfa6b60c755c7b03 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa33159a700 1 -- 172.21.15.40:0/600167138 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32400e070 con 0x7fa3340b1460 2023-09-30T00:52:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa33159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa33159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:52:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa33159a700 10 monclient: dump: 2023-09-30T00:52:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:42.141 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:42.142 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa33159a700 10 monclient: _finish_auth 0 2023-09-30T00:52:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa33159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa33159a700 1 -- 172.21.15.40:0/600167138 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32400ba40 con 0x7fa3340b1460 2023-09-30T00:52:42.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa33159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa33159a700 1 -- 172.21.15.40:0/600167138 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3240092e0 con 0x7fa3340b1460 2023-09-30T00:52:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa3389e3700 5 monclient: authenticate success, global_id 4746 2023-09-30T00:52:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa3389e3700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa33159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa3389e3700 4 set_mon_vals no callback set 2023-09-30T00:52:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa33159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:52:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa33159a700 10 monclient: dump: 2023-09-30T00:52:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:42.143 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:42.144 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa3389e3700 10 monclient: shutdown 2023-09-30T00:52:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa3389e3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa3389e3700 1 -- 172.21.15.40:0/600167138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3340b1460 msgr2=0x7fa3340b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.140+0000 7fa3389e3700 1 --2- 172.21.15.40:0/600167138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3340b1460 0x7fa3340b1830 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa324009f20 tx=0x7fa324009840 comp rx=0 tx=0).stop 2023-09-30T00:52:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.141+0000 7fa3389e3700 1 -- 172.21.15.40:0/600167138 shutdown_connections 2023-09-30T00:52:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.141+0000 7fa3389e3700 1 --2- 172.21.15.40:0/600167138 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa33413ac20 0x7fa33413b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:42.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.141+0000 7fa3389e3700 1 --2- 172.21.15.40:0/600167138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3340b1460 0x7fa3340b1830 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.141+0000 7fa3389e3700 1 -- 172.21.15.40:0/600167138 >> 172.21.15.40:0/600167138 conn(0x7fa3340631f0 msgr2=0x7fa3340635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.141+0000 7fa3389e3700 1 -- 172.21.15.40:0/600167138 shutdown_connections 2023-09-30T00:52:42.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.141+0000 7fa3389e3700 1 -- 172.21.15.40:0/600167138 wait complete. 2023-09-30T00:52:42.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.141+0000 7fa3389e3700 10 monclient: build_initial_monmap 2023-09-30T00:52:42.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.141+0000 7fa3389e3700 10 monclient: monmap: 2023-09-30T00:52:42.147 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:42.147 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:42.147 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:42.147 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:42.147 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:42.147 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:42.147 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:42.147 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:42.147 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:42.147 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:42.147 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 1 librados: starting msgr at 2023-09-30T00:52:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 1 librados: starting objecter 2023-09-30T00:52:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 1 Processor -- start 2023-09-30T00:52:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 1 -- start start 2023-09-30T00:52:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 1 librados: setting wanted keys 2023-09-30T00:52:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 1 librados: calling monclient init 2023-09-30T00:52:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 10 monclient: init 2023-09-30T00:52:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:52:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3340a5fb0 0x7fa33413df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 10 monclient(hunting): picked mon.noname-b con 0x7fa3340a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:42.148 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3340b1460 0x7fa3340a9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 10 monclient(hunting): picked mon.noname-c con 0x7fa3340b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa33413ac20 0x7fa334139c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 10 monclient(hunting): picked mon.noname-a con 0x7fa33413ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3340a9dc0 con 0x7fa33413ac20 2023-09-30T00:52:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3340a9aa0 con 0x7fa3340a5fb0 2023-09-30T00:52:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33413eb40 con 0x7fa3340b1460 2023-09-30T00:52:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.142+0000 7fa3389e3700 10 monclient(hunting): authenticate will time out at 5623.443967s 2023-09-30T00:52:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.144+0000 7fa332d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa33413ac20 0x7fa334139c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:42.149 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.146+0000 7fa332d9d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa33413ac20 0x7fa334139c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60804/0 (socket says 172.21.15.40:60804) 2023-09-30T00:52:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.146+0000 7fa332d9d700 1 -- 172.21.15.40:0/2216833740 learned_addr learned my addr 172.21.15.40:0/2216833740 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.146+0000 7fa332d9d700 10 monclient(hunting): get_auth_request con 0x7fa33413ac20 auth_method 0 2023-09-30T00:52:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.146+0000 7fa332d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.146+0000 7fa332d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:42.150 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.146+0000 7fa332d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:42.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.146+0000 7fa33259c700 1 --2- 172.21.15.40:0/2216833740 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3340a5fb0 0x7fa33413df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:42.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.147+0000 7fa332d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:42.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.147+0000 7fa332d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:42.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.147+0000 7fa332d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:42.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.147+0000 7fa33259c700 10 monclient(hunting): get_auth_request con 0x7fa3340a5fb0 auth_method 0 2023-09-30T00:52:42.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.147+0000 7fa33259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:42.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.147+0000 7fa33259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:42.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.147+0000 7fa33259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:42.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.148+0000 7fa33259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:42.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.148+0000 7fa33259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:42.151 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.148+0000 7fa33259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:42.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.151+0000 7fa332d9d700 10 monclient(hunting): handle_auth_done global_id 4749 payload 824 2023-09-30T00:52:42.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.151+0000 7fa332d9d700 1 -- 172.21.15.40:0/2216833740 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3340b1460 msgr2=0x7fa3340a9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:42.152 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.151+0000 7fa332d9d700 1 --2- 172.21.15.40:0/2216833740 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3340b1460 0x7fa3340a9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:42.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.151+0000 7fa332d9d700 1 -- 172.21.15.40:0/2216833740 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3340a5fb0 msgr2=0x7fa33413df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:42.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.151+0000 7fa332d9d700 1 --2- 172.21.15.40:0/2216833740 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3340a5fb0 0x7fa33413df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:42.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.151+0000 7fa332d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:52:42.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.151+0000 7fa332d9d700 1 monclient: found mon.noname-a 2023-09-30T00:52:42.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.151+0000 7fa332d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:42.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.151+0000 7fa332d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:52:42.153 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.151+0000 7fa332d9d700 1 -- 172.21.15.40:0/2216833740 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa324009ae0 con 0x7fa33413ac20 2023-09-30T00:52:42.154 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.154+0000 7fa332d9d700 1 --2- 172.21.15.40:0/2216833740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa33413ac20 0x7fa334139c20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00b590 tx=0x7fa32c006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:42.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.154+0000 7fa32a7fc700 10 client.?.objecter ms_handle_connect 0x7fa33413ac20 2023-09-30T00:52:42.155 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.154+0000 7fa32a7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:42.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa32a7fc700 1 -- 172.21.15.40:0/2216833740 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c00f5e0 con 0x7fa33413ac20 2023-09-30T00:52:42.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa32a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:42.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa32a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:52:42.161 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa32a7fc700 10 monclient: dump: 2023-09-30T00:52:42.161 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:42.161 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:42.161 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa32a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa32a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:42.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa32a7fc700 1 -- 172.21.15.40:0/2216833740 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32c00f740 con 0x7fa33413ac20 2023-09-30T00:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa3389e3700 5 monclient: authenticate success, global_id 4749 2023-09-30T00:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa3389e3700 10 monclient: _renew_subs 2023-09-30T00:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa3389e3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa3389e3700 1 -- 172.21.15.40:0/2216833740 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33413a160 con 0x7fa33413ac20 2023-09-30T00:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa32a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa32a7fc700 1 -- 172.21.15.40:0/2216833740 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c00fb00 con 0x7fa33413ac20 2023-09-30T00:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa32a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa32a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa32a7fc700 10 monclient: dump: 2023-09-30T00:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:42.163 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa330d99700 4 set_mon_vals no callback set 2023-09-30T00:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa3389e3700 10 client.4749.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa3389e3700 10 monclient: _renew_subs 2023-09-30T00:52:42.164 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa3389e3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa3389e3700 1 -- 172.21.15.40:0/2216833740 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33413a650 con 0x7fa33413ac20 2023-09-30T00:52:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.160+0000 7fa3389e3700 1 librados: init done 2023-09-30T00:52:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.161+0000 7fa3389e3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.161+0000 7fa3389e3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:42.165 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.161+0000 7fa3389e3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:42.166 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.161+0000 7fa3389e3700 1 -- 172.21.15.40:0/2216833740 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa334064a00 con 0x7fa33413ac20 2023-09-30T00:52:42.188 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.187+0000 7fa32a7fc700 1 -- 172.21.15.40:0/2216833740 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa32c024430 con 0x7fa33413ac20 2023-09-30T00:52:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.188+0000 7fa32a7fc700 1 --2- 172.21.15.40:0/2216833740 >> v2:172.21.15.135:6800/116426 conn(0x7fa314068940 0x7fa31406adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.188+0000 7fa32a7fc700 1 -- 172.21.15.40:0/2216833740 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa32c084580 con 0x7fa33413ac20 2023-09-30T00:52:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.188+0000 7fa32a7fc700 10 client.4749.objecter ms_dispatch 0x7fa334001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.188+0000 7fa32a7fc700 3 client.4749.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.188+0000 7fa32a7fc700 3 client.4749.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.188+0000 7fa32a7fc700 20 client.4749.objecter dump_active .. 0 homeless 2023-09-30T00:52:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.188+0000 7fa33259c700 1 --2- 172.21.15.40:0/2216833740 >> v2:172.21.15.135:6800/116426 conn(0x7fa314068940 0x7fa31406adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:42.189 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.188+0000 7fa33259c700 10 monclient: get_auth_request con 0x7fa314068940 auth_method 0 2023-09-30T00:52:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.189+0000 7fa33259c700 1 --2- 172.21.15.40:0/2216833740 >> v2:172.21.15.135:6800/116426 conn(0x7fa314068940 0x7fa31406adf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa324009880 tx=0x7fa324017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:42.190 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.189+0000 7fa32a7fc700 10 client.4749.objecter ms_handle_connect 0x7fa314068940 2023-09-30T00:52:42.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.464+0000 7fa32a7fc700 1 -- 172.21.15.40:0/2216833740 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa32c051ea0 con 0x7fa33413ac20 2023-09-30T00:52:42.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.464+0000 7fa32a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:42.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.464+0000 7fa32a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:42.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.722+0000 7fa3389e3700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:42.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.722+0000 7fa3389e3700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:42.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.722+0000 7fa3389e3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:42.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.722+0000 7fa3389e3700 1 -- 172.21.15.40:0/2216833740 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa3340a56d0 con 0x7fa33413ac20 2023-09-30T00:52:42.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.730+0000 7fa32a7fc700 1 -- 172.21.15.40:0/2216833740 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa32c0554c0 con 0x7fa33413ac20 2023-09-30T00:52:42.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.730+0000 7fa32a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:42.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.730+0000 7fa32a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:42.731 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:42.731 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.732+0000 7fa3389e3700 10 librados: watch_flush enter 2023-09-30T00:52:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 10 librados: watch_flush exit 2023-09-30T00:52:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 20 client.4749.objecter shutdown clearing up homeless session... 2023-09-30T00:52:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 10 client.4749.objecter successfully canceled tick 2023-09-30T00:52:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 1 -- 172.21.15.40:0/2216833740 >> v2:172.21.15.135:6800/116426 conn(0x7fa314068940 msgr2=0x7fa31406adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 1 --2- 172.21.15.40:0/2216833740 >> v2:172.21.15.135:6800/116426 conn(0x7fa314068940 0x7fa31406adf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa324009880 tx=0x7fa324017000 comp rx=0 tx=0).stop 2023-09-30T00:52:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 10 monclient: shutdown 2023-09-30T00:52:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 1 -- 172.21.15.40:0/2216833740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa33413ac20 msgr2=0x7fa334139c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:42.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 1 --2- 172.21.15.40:0/2216833740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa33413ac20 0x7fa334139c20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00b590 tx=0x7fa32c006f90 comp rx=0 tx=0).stop 2023-09-30T00:52:42.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 1 -- 172.21.15.40:0/2216833740 shutdown_connections 2023-09-30T00:52:42.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 1 --2- 172.21.15.40:0/2216833740 >> v2:172.21.15.135:6800/116426 conn(0x7fa314068940 0x7fa31406adf0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:42.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 1 --2- 172.21.15.40:0/2216833740 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa3340a5fb0 0x7fa33413df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:42.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 1 --2- 172.21.15.40:0/2216833740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa33413ac20 0x7fa334139c20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:42.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 1 --2- 172.21.15.40:0/2216833740 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa3340b1460 0x7fa3340a9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:42.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 1 -- 172.21.15.40:0/2216833740 >> 172.21.15.40:0/2216833740 conn(0x7fa3340631f0 msgr2=0x7fa3340b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:42.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 1 -- 172.21.15.40:0/2216833740 shutdown_connections 2023-09-30T00:52:42.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 1 -- 172.21.15.40:0/2216833740 wait complete. 2023-09-30T00:52:42.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:42.733+0000 7fa3389e3700 1 librados: shutdown 2023-09-30T00:52:45.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.887+0000 7f96355b6700 10 monclient: get_monmap_and_config 2023-09-30T00:52:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.887+0000 7f96355b6700 10 monclient: build_initial_monmap 2023-09-30T00:52:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.887+0000 7f96355b6700 10 monclient: monmap: 2023-09-30T00:52:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:45.888532+0000 2023-09-30T00:52:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:45.888532+0000 2023-09-30T00:52:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:45.889 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:45.889 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:45.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 1 Processor -- start 2023-09-30T00:52:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 1 -- start start 2023-09-30T00:52:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 10 monclient: init 2023-09-30T00:52:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-09-30T00:52:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f96300a5710 0x7f96300a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 10 monclient(hunting): picked mon.noname-a con 0x7f96300a5710 addr v2:172.21.15.40:3300/0 2023-09-30T00:52:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96300a7b20 addr v1:172.21.15.40:6789/0 2023-09-30T00:52:45.891 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f963013e8e0 addr v1:172.21.15.135:6789/0 2023-09-30T00:52:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96300a7ff0 con 0x7f96300a7b20 2023-09-30T00:52:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96300aedc0 con 0x7f963013e8e0 2023-09-30T00:52:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96300aef50 con 0x7f96300a5710 2023-09-30T00:52:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.888+0000 7f96355b6700 10 monclient(hunting): authenticate will time out at 5627.189954s 2023-09-30T00:52:45.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.889+0000 7f962effd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f96300a5710 0x7f96300a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:45.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.889+0000 7f962f7fe700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f963013e8e0 0x7f963013ae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:38856/0 (socket says 172.21.15.40:38856) 2023-09-30T00:52:45.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.889+0000 7f962f7fe700 1 -- 172.21.15.40:0/4260870331 learned_addr learned my addr 172.21.15.40:0/4260870331 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:52:45.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.890+0000 7f962effd700 10 monclient(hunting): get_auth_request con 0x7f96300a5710 auth_method 0 2023-09-30T00:52:45.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.890+0000 7f962effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:45.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.890+0000 7f962effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:45.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.890+0000 7f962effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:45.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.891+0000 7f962effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:45.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.891+0000 7f962effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:45.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.891+0000 7f962effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:45.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.892+0000 7f962dffb700 1 -- 172.21.15.40:0/4260870331 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707711740 0 0) 0x7f96300aedc0 con 0x7f963013e8e0 2023-09-30T00:52:45.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.892+0000 7f962dffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:45.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.892+0000 7f962dffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:45.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.892+0000 7f962dffb700 10 monclient(hunting): my global_id is 4828 2023-09-30T00:52:45.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.892+0000 7f962dffb700 1 -- 172.21.15.40:0/4260870331 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9618003570 con 0x7f963013e8e0 2023-09-30T00:52:45.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.894+0000 7f962effd700 10 monclient(hunting): handle_auth_done global_id 4752 payload 274 2023-09-30T00:52:45.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.894+0000 7f962effd700 1 -- 172.21.15.40:0/4260870331 >> v1:172.21.15.135:6789/0 conn(0x7f963013e8e0 legacy=0x7f963013ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:45.896 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.894+0000 7f962effd700 1 -- 172.21.15.40:0/4260870331 >> v1:172.21.15.40:6789/0 conn(0x7f96300a7b20 legacy=0x7f963013e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:45.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.894+0000 7f962effd700 10 monclient: _finish_hunting 0 2023-09-30T00:52:45.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.894+0000 7f962effd700 1 monclient: found mon.noname-a 2023-09-30T00:52:45.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.894+0000 7f962effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:45.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.894+0000 7f962effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:52:45.897 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.894+0000 7f962effd700 1 -- 172.21.15.40:0/4260870331 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963013ed80 con 0x7f96300a5710 2023-09-30T00:52:45.898 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.896+0000 7f962effd700 1 --2- 172.21.15.40:0/4260870331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96300a5710 0x7f96300a75e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f96200098e0 tx=0x7f96200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f55e1a3994a2cef1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f962dffb700 1 -- 172.21.15.40:0/4260870331 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962000b640 con 0x7f96300a5710 2023-09-30T00:52:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f962dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f962dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:52:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f962dffb700 10 monclient: dump: 2023-09-30T00:52:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:45.906 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:45.906 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:45.906 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:45.906 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:45.906 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:45.906 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:45.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f962dffb700 10 monclient: _finish_auth 0 2023-09-30T00:52:45.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f962dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:45.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f962dffb700 1 -- 172.21.15.40:0/4260870331 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962000b7a0 con 0x7f96300a5710 2023-09-30T00:52:45.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f96355b6700 5 monclient: authenticate success, global_id 4752 2023-09-30T00:52:45.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f96355b6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:52:45.907 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f962dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f96355b6700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f962dffb700 1 -- 172.21.15.40:0/4260870331 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962000bb40 con 0x7f96300a5710 2023-09-30T00:52:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f96355b6700 4 set_mon_vals no callback set 2023-09-30T00:52:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f962dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f962dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:52:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.903+0000 7f962dffb700 10 monclient: dump: 2023-09-30T00:52:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:45.910 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:45.910 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:45.910 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.904+0000 7f96355b6700 10 monclient: shutdown 2023-09-30T00:52:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.904+0000 7f96355b6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:45.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.904+0000 7f96355b6700 1 -- 172.21.15.40:0/4260870331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96300a5710 msgr2=0x7f96300a75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:45.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.904+0000 7f96355b6700 1 --2- 172.21.15.40:0/4260870331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96300a5710 0x7f96300a75e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f96200098e0 tx=0x7f96200091a0 comp rx=0 tx=0).stop 2023-09-30T00:52:45.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.904+0000 7f96355b6700 1 -- 172.21.15.40:0/4260870331 shutdown_connections 2023-09-30T00:52:45.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.904+0000 7f96355b6700 1 --2- 172.21.15.40:0/4260870331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96300a5710 0x7f96300a75e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:45.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.904+0000 7f96355b6700 1 -- 172.21.15.40:0/4260870331 >> 172.21.15.40:0/4260870331 conn(0x7f9630069080 msgr2=0x7f96300631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:45.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.904+0000 7f96355b6700 1 -- 172.21.15.40:0/4260870331 shutdown_connections 2023-09-30T00:52:45.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.904+0000 7f96355b6700 1 -- 172.21.15.40:0/4260870331 wait complete. 2023-09-30T00:52:45.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.905+0000 7f96355b6700 10 monclient: build_initial_monmap 2023-09-30T00:52:45.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.905+0000 7f96355b6700 10 monclient: monmap: 2023-09-30T00:52:45.917 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:45.918 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:45.918 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:45.918 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:45.918 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:45.918 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:45.918 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:45.918 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:45.919 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.906+0000 7f96355b6700 1 librados: starting msgr at 2023-09-30T00:52:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.906+0000 7f96355b6700 1 librados: starting objecter 2023-09-30T00:52:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.906+0000 7f96355b6700 1 Processor -- start 2023-09-30T00:52:45.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.906+0000 7f96355b6700 1 -- start start 2023-09-30T00:52:45.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.906+0000 7f96355b6700 1 librados: setting wanted keys 2023-09-30T00:52:45.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.906+0000 7f96355b6700 1 librados: calling monclient init 2023-09-30T00:52:45.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.906+0000 7f96355b6700 10 monclient: init 2023-09-30T00:52:45.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:45.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:52:45.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96300a5710 0x7f96300ad9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:45.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 10 monclient(hunting): picked mon.noname-a con 0x7f96300a5710 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:45.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96300a7b20 0x7f96300aa7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:45.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 10 monclient(hunting): picked mon.noname-c con 0x7f96300a7b20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:45.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f963013e8e0 0x7f96300b4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:45.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 10 monclient(hunting): picked mon.noname-b con 0x7f963013e8e0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:45.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:45.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300b3a50 con 0x7f96300a5710 2023-09-30T00:52:45.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:45.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300ac8c0 con 0x7f963013e8e0 2023-09-30T00:52:45.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:45.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96300ad480 con 0x7f96300a7b20 2023-09-30T00:52:45.922 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:45.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f96355b6700 10 monclient(hunting): authenticate will time out at 5627.208174s 2023-09-30T00:52:45.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f962e7fc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96300a7b20 0x7f96300aa7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:45.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.907+0000 7f962f7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f963013e8e0 0x7f96300b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:45.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96300a5710 0x7f96300ad9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:45.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962e7fc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96300a7b20 0x7f96300aa7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:46390/0 (socket says 172.21.15.40:46390) 2023-09-30T00:52:45.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962e7fc700 1 -- 172.21.15.40:0/4289116739 learned_addr learned my addr 172.21.15.40:0/4289116739 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:45.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962e7fc700 10 monclient(hunting): get_auth_request con 0x7f96300a7b20 auth_method 0 2023-09-30T00:52:45.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:45.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:45.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:45.923 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962f7fe700 10 monclient(hunting): get_auth_request con 0x7f963013e8e0 auth_method 0 2023-09-30T00:52:45.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:45.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:45.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:45.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962effd700 10 monclient(hunting): get_auth_request con 0x7f96300a5710 auth_method 0 2023-09-30T00:52:45.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:45.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:45.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.908+0000 7f962effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:45.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.909+0000 7f962f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:45.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.909+0000 7f962f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:45.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.909+0000 7f962f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:45.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.909+0000 7f962e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:45.924 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.909+0000 7f962e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:45.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.909+0000 7f962e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:45.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.909+0000 7f962effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:45.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.909+0000 7f962effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:45.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.909+0000 7f962effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:45.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.912+0000 7f962f7fe700 10 monclient(hunting): handle_auth_done global_id 4831 payload 824 2023-09-30T00:52:45.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.912+0000 7f962f7fe700 1 -- 172.21.15.40:0/4289116739 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96300a7b20 msgr2=0x7f96300aa7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:45.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.912+0000 7f962f7fe700 1 --2- 172.21.15.40:0/4289116739 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96300a7b20 0x7f96300aa7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:45.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.912+0000 7f962f7fe700 1 -- 172.21.15.40:0/4289116739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96300a5710 msgr2=0x7f96300ad9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:45.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.912+0000 7f962f7fe700 1 --2- 172.21.15.40:0/4289116739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96300a5710 0x7f96300ad9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:45.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.912+0000 7f962f7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:52:45.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.912+0000 7f962f7fe700 1 monclient: found mon.noname-b 2023-09-30T00:52:45.925 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.912+0000 7f962f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:45.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.912+0000 7f962f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:52:45.926 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.912+0000 7f962f7fe700 1 -- 172.21.15.40:0/4289116739 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9620009420 con 0x7f963013e8e0 2023-09-30T00:52:45.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.915+0000 7f962f7fe700 1 --2- 172.21.15.40:0/4289116739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f963013e8e0 0x7f96300b4520 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f962800cb90 tx=0x7f9628001980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:45.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.915+0000 7f9616ffd700 10 client.?.objecter ms_handle_connect 0x7f963013e8e0 2023-09-30T00:52:45.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.915+0000 7f9616ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:45.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f9616ffd700 1 -- 172.21.15.40:0/4289116739 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962800a7e0 con 0x7f963013e8e0 2023-09-30T00:52:45.927 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f9616ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:45.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f9616ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:52:45.928 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f9616ffd700 10 monclient: dump: 2023-09-30T00:52:45.928 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:45.928 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:45.928 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:45.928 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:45.928 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:45.928 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:45.929 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:45.929 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:45.929 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:45.929 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:45.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f9616ffd700 10 monclient: _finish_auth 0 2023-09-30T00:52:45.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f9616ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:45.929 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f9616ffd700 1 -- 172.21.15.40:0/4289116739 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962800a940 con 0x7f963013e8e0 2023-09-30T00:52:45.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f9616ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:45.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f9616ffd700 1 -- 172.21.15.40:0/4289116739 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962800acc0 con 0x7f963013e8e0 2023-09-30T00:52:45.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f9616ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:45.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f9616ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:52:45.930 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f9616ffd700 10 monclient: dump: 2023-09-30T00:52:45.930 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:45.930 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:45.931 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:45.931 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:45.931 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:45.931 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:45.931 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:45.931 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:45.931 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:45.932 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:45.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f96355b6700 5 monclient: authenticate success, global_id 4831 2023-09-30T00:52:45.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f96355b6700 10 monclient: _renew_subs 2023-09-30T00:52:45.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f962d7fa700 4 set_mon_vals no callback set 2023-09-30T00:52:45.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f96355b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:45.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f96355b6700 1 -- 172.21.15.40:0/4289116739 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96300aace0 con 0x7f963013e8e0 2023-09-30T00:52:45.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f96355b6700 10 client.4831.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:45.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f96355b6700 10 monclient: _renew_subs 2023-09-30T00:52:45.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f96355b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:45.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f96355b6700 1 -- 172.21.15.40:0/4289116739 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96300aeb80 con 0x7f963013e8e0 2023-09-30T00:52:45.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.920+0000 7f96355b6700 1 librados: init done 2023-09-30T00:52:45.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.921+0000 7f96355b6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:45.932 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.921+0000 7f96355b6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:45.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.921+0000 7f96355b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:45.933 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:45.921+0000 7f96355b6700 1 -- 172.21.15.40:0/4289116739 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96300ad6a0 con 0x7f963013e8e0 2023-09-30T00:52:46.175 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.174+0000 7f9616ffd700 1 -- 172.21.15.40:0/4289116739 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9628007300 con 0x7f963013e8e0 2023-09-30T00:52:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.175+0000 7f9616ffd700 1 --2- 172.21.15.40:0/4289116739 >> v2:172.21.15.135:6800/116426 conn(0x7f961c068960 0x7f961c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.175+0000 7f962effd700 1 --2- 172.21.15.40:0/4289116739 >> v2:172.21.15.135:6800/116426 conn(0x7f961c068960 0x7f961c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.175+0000 7f962effd700 10 monclient: get_auth_request con 0x7f961c068960 auth_method 0 2023-09-30T00:52:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.176+0000 7f962effd700 1 --2- 172.21.15.40:0/4289116739 >> v2:172.21.15.135:6800/116426 conn(0x7f961c068960 0x7f961c06ae10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f96200093f0 tx=0x7f9620017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:46.176 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.176+0000 7f9616ffd700 10 client.4831.objecter ms_handle_connect 0x7f961c068960 2023-09-30T00:52:46.180 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.180+0000 7f9616ffd700 1 -- 172.21.15.40:0/4289116739 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f962800ae20 con 0x7f963013e8e0 2023-09-30T00:52:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.180+0000 7f9616ffd700 10 client.4831.objecter ms_dispatch 0x7f9630001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.180+0000 7f9616ffd700 3 client.4831.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.180+0000 7f9616ffd700 3 client.4831.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:46.181 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.180+0000 7f9616ffd700 20 client.4831.objecter dump_active .. 0 homeless 2023-09-30T00:52:46.190 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.189+0000 7f9616ffd700 1 -- 172.21.15.40:0/4289116739 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9628051c20 con 0x7f963013e8e0 2023-09-30T00:52:46.190 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.189+0000 7f9616ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:46.190 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.189+0000 7f9616ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:46.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.447+0000 7f96355b6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.447+0000 7f96355b6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.447+0000 7f96355b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:46.448 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.447+0000 7f96355b6700 1 -- 172.21.15.40:0/4289116739 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f96300aca00 con 0x7f963013e8e0 2023-09-30T00:52:46.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.452+0000 7f9616ffd700 1 -- 172.21.15.40:0/4289116739 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f96300aca00 con 0x7f963013e8e0 2023-09-30T00:52:46.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.452+0000 7f9616ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:46.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.452+0000 7f9616ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:46.453 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:46.453 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:46.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 10 librados: watch_flush enter 2023-09-30T00:52:46.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 10 librados: watch_flush exit 2023-09-30T00:52:46.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 20 client.4831.objecter shutdown clearing up homeless session... 2023-09-30T00:52:46.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 10 client.4831.objecter successfully canceled tick 2023-09-30T00:52:46.455 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 1 -- 172.21.15.40:0/4289116739 >> v2:172.21.15.135:6800/116426 conn(0x7f961c068960 msgr2=0x7f961c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:46.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 1 --2- 172.21.15.40:0/4289116739 >> v2:172.21.15.135:6800/116426 conn(0x7f961c068960 0x7f961c06ae10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f96200093f0 tx=0x7f9620017000 comp rx=0 tx=0).stop 2023-09-30T00:52:46.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 10 monclient: shutdown 2023-09-30T00:52:46.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:46.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 1 -- 172.21.15.40:0/4289116739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f963013e8e0 msgr2=0x7f96300b4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:46.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 1 --2- 172.21.15.40:0/4289116739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f963013e8e0 0x7f96300b4520 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f962800cb90 tx=0x7f9628001980 comp rx=0 tx=0).stop 2023-09-30T00:52:46.456 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 1 -- 172.21.15.40:0/4289116739 shutdown_connections 2023-09-30T00:52:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 1 --2- 172.21.15.40:0/4289116739 >> v2:172.21.15.135:6800/116426 conn(0x7f961c068960 0x7f961c06ae10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 1 --2- 172.21.15.40:0/4289116739 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f963013e8e0 0x7f96300b4520 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 1 --2- 172.21.15.40:0/4289116739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96300a5710 0x7f96300ad9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 1 --2- 172.21.15.40:0/4289116739 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f96300a7b20 0x7f96300aa7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 1 -- 172.21.15.40:0/4289116739 >> 172.21.15.40:0/4289116739 conn(0x7f9630069080 msgr2=0x7f9630136fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 1 -- 172.21.15.40:0/4289116739 shutdown_connections 2023-09-30T00:52:46.457 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 1 -- 172.21.15.40:0/4289116739 wait complete. 2023-09-30T00:52:46.458 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:46.454+0000 7f96355b6700 1 librados: shutdown 2023-09-30T00:52:49.470 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:49.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.606+0000 7f097d3ec700 10 monclient: get_monmap_and_config 2023-09-30T00:52:49.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.606+0000 7f097d3ec700 10 monclient: build_initial_monmap 2023-09-30T00:52:49.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.606+0000 7f097d3ec700 10 monclient: monmap: 2023-09-30T00:52:49.607 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:49.607 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:49.607 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:49.607110+0000 2023-09-30T00:52:49.607 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:49.607110+0000 2023-09-30T00:52:49.607 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:49.607 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:49.607 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:49.607 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:49.608 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:49.608 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:49.608 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:49.608 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:49.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.606+0000 7f097d3ec700 1 Processor -- start 2023-09-30T00:52:49.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.606+0000 7f097d3ec700 1 -- start start 2023-09-30T00:52:49.608 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.606+0000 7f097d3ec700 10 monclient: init 2023-09-30T00:52:49.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.607+0000 7f097d3ec700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:49.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.607+0000 7f097d3ec700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2023-09-30T00:52:49.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.607+0000 7f097d3ec700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f09780b1460 0x7f09780b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:49.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.607+0000 7f097d3ec700 10 monclient(hunting): picked mon.noname-a con 0x7f09780b1460 addr v2:172.21.15.40:3300/0 2023-09-30T00:52:49.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.607+0000 7f097d3ec700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09780a5fb0 addr v1:172.21.15.40:6789/0 2023-09-30T00:52:49.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.607+0000 7f097d3ec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097813ac20 0x7f097813b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:49.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.607+0000 7f097d3ec700 10 monclient(hunting): picked mon.noname-b con 0x7f097813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:49.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.607+0000 7f097d3ec700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09780a5b20 con 0x7f09780a5fb0 2023-09-30T00:52:49.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.607+0000 7f097d3ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:49.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.607+0000 7f097d3ec700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09780a71f0 con 0x7f09780b1460 2023-09-30T00:52:49.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.607+0000 7f097d3ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:49.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.607+0000 7f097d3ec700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09780a74d0 con 0x7f097813ac20 2023-09-30T00:52:49.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.607+0000 7f097d3ec700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:49.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.607+0000 7f097d3ec700 10 monclient(hunting): authenticate will time out at 5630.908941s 2023-09-30T00:52:49.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.608+0000 7f09777fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097813ac20 0x7f097813b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:49.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.608+0000 7f0976ffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f09780b1460 0x7f09780b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:49.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.608+0000 7f09777fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097813ac20 0x7f097813b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:46396/0 (socket says 172.21.15.40:46396) 2023-09-30T00:52:49.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.608+0000 7f09777fe700 1 -- 172.21.15.40:0/1579974604 learned_addr learned my addr 172.21.15.40:0/1579974604 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:49.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.608+0000 7f09777fe700 10 monclient(hunting): get_auth_request con 0x7f097813ac20 auth_method 0 2023-09-30T00:52:49.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.608+0000 7f09777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:49.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.608+0000 7f09777fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:49.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.608+0000 7f09777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:49.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.609+0000 7f0976ffd700 10 monclient(hunting): get_auth_request con 0x7f09780b1460 auth_method 0 2023-09-30T00:52:49.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.609+0000 7f0976ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:49.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.609+0000 7f0976ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:49.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.609+0000 7f0976ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:49.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.609+0000 7f09777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:49.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.609+0000 7f09777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:49.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.609+0000 7f09777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:49.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.610+0000 7f0976ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:49.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.610+0000 7f0976ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:49.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.610+0000 7f0976ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:49.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.612+0000 7f09777fe700 10 monclient(hunting): handle_auth_done global_id 4658 payload 274 2023-09-30T00:52:49.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.612+0000 7f09777fe700 1 -- 172.21.15.40:0/1579974604 >> v2:172.21.15.40:3300/0 conn(0x7f09780b1460 msgr2=0x7f09780b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:52:49.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.612+0000 7f09777fe700 1 --2- 172.21.15.40:0/1579974604 >> v2:172.21.15.40:3300/0 conn(0x7f09780b1460 0x7f09780b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:49.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.612+0000 7f09777fe700 1 -- 172.21.15.40:0/1579974604 >> v1:172.21.15.40:6789/0 conn(0x7f09780a5fb0 legacy=0x7f0978141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:49.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.612+0000 7f09777fe700 10 monclient: _finish_hunting 0 2023-09-30T00:52:49.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.612+0000 7f09777fe700 1 monclient: found mon.noname-b 2023-09-30T00:52:49.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.612+0000 7f09777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:49.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.612+0000 7f09777fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:52:49.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.612+0000 7f09777fe700 1 -- 172.21.15.40:0/1579974604 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09780a63c0 con 0x7f097813ac20 2023-09-30T00:52:49.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.614+0000 7f09777fe700 1 --2- 172.21.15.40:0/1579974604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097813ac20 0x7f097813b080 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0970009f20 tx=0x7f0970009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfc6f99b8141ede server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:49.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f0975ffb700 1 -- 172.21.15.40:0/1579974604 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f097000e070 con 0x7f097813ac20 2023-09-30T00:52:49.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f0975ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:49.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f0975ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:52:49.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f0975ffb700 10 monclient: dump: 2023-09-30T00:52:49.620 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:49.621 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f0975ffb700 10 monclient: _finish_auth 0 2023-09-30T00:52:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f0975ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f0975ffb700 1 -- 172.21.15.40:0/1579974604 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097000b9d0 con 0x7f097813ac20 2023-09-30T00:52:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f097d3ec700 5 monclient: authenticate success, global_id 4658 2023-09-30T00:52:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f097d3ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:52:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f0975ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f097d3ec700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f0975ffb700 1 -- 172.21.15.40:0/1579974604 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700092e0 con 0x7f097813ac20 2023-09-30T00:52:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f097d3ec700 4 set_mon_vals no callback set 2023-09-30T00:52:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f0975ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f0975ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:52:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.619+0000 7f0975ffb700 10 monclient: dump: 2023-09-30T00:52:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:49.623 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:49.623 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:49.623 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:49.623 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:49.623 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:49.623 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:49.623 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:49.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.620+0000 7f097d3ec700 10 monclient: shutdown 2023-09-30T00:52:49.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.620+0000 7f097d3ec700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:49.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.620+0000 7f097d3ec700 1 -- 172.21.15.40:0/1579974604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097813ac20 msgr2=0x7f097813b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.620+0000 7f097d3ec700 1 --2- 172.21.15.40:0/1579974604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097813ac20 0x7f097813b080 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0970009f20 tx=0x7f0970009840 comp rx=0 tx=0).stop 2023-09-30T00:52:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.620+0000 7f097d3ec700 1 -- 172.21.15.40:0/1579974604 shutdown_connections 2023-09-30T00:52:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.620+0000 7f097d3ec700 1 --2- 172.21.15.40:0/1579974604 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f097813ac20 0x7f097813b080 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.620+0000 7f097d3ec700 1 --2- 172.21.15.40:0/1579974604 >> v2:172.21.15.40:3300/0 conn(0x7f09780b1460 0x7f09780b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.620+0000 7f097d3ec700 1 -- 172.21.15.40:0/1579974604 >> 172.21.15.40:0/1579974604 conn(0x7f09780631f0 msgr2=0x7f09780635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.620+0000 7f097d3ec700 1 -- 172.21.15.40:0/1579974604 shutdown_connections 2023-09-30T00:52:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.620+0000 7f097d3ec700 1 -- 172.21.15.40:0/1579974604 wait complete. 2023-09-30T00:52:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.621+0000 7f097d3ec700 10 monclient: build_initial_monmap 2023-09-30T00:52:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.621+0000 7f097d3ec700 10 monclient: monmap: 2023-09-30T00:52:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:49.627 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:49.628 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 1 librados: starting msgr at 2023-09-30T00:52:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 1 librados: starting objecter 2023-09-30T00:52:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 1 Processor -- start 2023-09-30T00:52:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 1 -- start start 2023-09-30T00:52:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 1 librados: setting wanted keys 2023-09-30T00:52:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 1 librados: calling monclient init 2023-09-30T00:52:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 10 monclient: init 2023-09-30T00:52:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:52:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09780a5fb0 0x7f097813e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 10 monclient(hunting): picked mon.noname-b con 0x7f09780a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09780b1460 0x7f09780a79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 10 monclient(hunting): picked mon.noname-c con 0x7f09780b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f097813ac20 0x7f09780a7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 10 monclient(hunting): picked mon.noname-a con 0x7f097813ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09780ab960 con 0x7f097813ac20 2023-09-30T00:52:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.622+0000 7f097d3ec700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09780abc40 con 0x7f09780a5fb0 2023-09-30T00:52:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.623+0000 7f097d3ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.623+0000 7f097d3ec700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09780aa4f0 con 0x7f09780b1460 2023-09-30T00:52:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.623+0000 7f097d3ec700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.623+0000 7f097d3ec700 10 monclient(hunting): authenticate will time out at 5630.924017s 2023-09-30T00:52:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.623+0000 7f09777fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f097813ac20 0x7f09780a7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.623+0000 7f0976ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09780a5fb0 0x7f097813e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f09777fe700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f097813ac20 0x7f09780a7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60852/0 (socket says 172.21.15.40:60852) 2023-09-30T00:52:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f0976ffd700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09780a5fb0 0x7f097813e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:43494/0 (socket says 172.21.15.40:43494) 2023-09-30T00:52:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f09777fe700 1 -- 172.21.15.40:0/1627497596 learned_addr learned my addr 172.21.15.40:0/1627497596 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f09777fe700 10 monclient(hunting): get_auth_request con 0x7f097813ac20 auth_method 0 2023-09-30T00:52:49.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f09777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:49.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f09777fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:49.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f09777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:49.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f0976ffd700 10 monclient(hunting): get_auth_request con 0x7f09780a5fb0 auth_method 0 2023-09-30T00:52:49.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f0976ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:49.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f0976ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:49.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f0976ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:49.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f09777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:49.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f09777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:49.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f09777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:49.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f0976ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:49.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f0976ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:49.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f0976ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:49.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.624+0000 7f09767fc700 1 --2- 172.21.15.40:0/1627497596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09780b1460 0x7f09780a79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:49.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.625+0000 7f09767fc700 10 monclient(hunting): get_auth_request con 0x7f09780b1460 auth_method 0 2023-09-30T00:52:49.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.625+0000 7f09767fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:49.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.625+0000 7f09767fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:49.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.625+0000 7f09767fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:49.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.625+0000 7f09767fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:49.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.625+0000 7f09767fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:49.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.625+0000 7f09767fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.628+0000 7f09777fe700 10 monclient(hunting): handle_auth_done global_id 4764 payload 824 2023-09-30T00:52:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.628+0000 7f09777fe700 1 -- 172.21.15.40:0/1627497596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09780b1460 msgr2=0x7f09780a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.628+0000 7f09777fe700 1 --2- 172.21.15.40:0/1627497596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09780b1460 0x7f09780a79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.628+0000 7f09777fe700 1 -- 172.21.15.40:0/1627497596 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09780a5fb0 msgr2=0x7f097813e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.628+0000 7f09777fe700 1 --2- 172.21.15.40:0/1627497596 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09780a5fb0 0x7f097813e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.628+0000 7f09777fe700 10 monclient: _finish_hunting 0 2023-09-30T00:52:49.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.628+0000 7f09777fe700 1 monclient: found mon.noname-a 2023-09-30T00:52:49.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.628+0000 7f09777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:49.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.628+0000 7f09777fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:52:49.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.628+0000 7f09777fe700 1 -- 172.21.15.40:0/1627497596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0970009ae0 con 0x7f097813ac20 2023-09-30T00:52:49.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.631+0000 7f09777fe700 1 --2- 172.21.15.40:0/1627497596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f097813ac20 0x7f09780a7f30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f097000b610 tx=0x7f0970018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:49.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.631+0000 7f095effd700 10 client.?.objecter ms_handle_connect 0x7f097813ac20 2023-09-30T00:52:49.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.631+0000 7f095effd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:49.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.633+0000 7f095effd700 1 -- 172.21.15.40:0/1627497596 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0970008ad0 con 0x7f097813ac20 2023-09-30T00:52:49.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.633+0000 7f095effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:49.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.633+0000 7f095effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:52:49.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.633+0000 7f095effd700 10 monclient: dump: 2023-09-30T00:52:49.636 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:49.637 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.633+0000 7f095effd700 10 monclient: _finish_auth 0 2023-09-30T00:52:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.633+0000 7f095effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:49.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.633+0000 7f097d3ec700 5 monclient: authenticate success, global_id 4764 2023-09-30T00:52:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.633+0000 7f097d3ec700 10 monclient: _renew_subs 2023-09-30T00:52:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.633+0000 7f097d3ec700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.633+0000 7f097d3ec700 1 -- 172.21.15.40:0/1627497596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09780aa0f0 con 0x7f097813ac20 2023-09-30T00:52:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.634+0000 7f097d3ec700 10 client.4764.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.634+0000 7f097d3ec700 10 monclient: _renew_subs 2023-09-30T00:52:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.634+0000 7f097d3ec700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.634+0000 7f097d3ec700 1 -- 172.21.15.40:0/1627497596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09780aac50 con 0x7f097813ac20 2023-09-30T00:52:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.634+0000 7f097d3ec700 1 librados: init done 2023-09-30T00:52:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.634+0000 7f097d3ec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.634+0000 7f097d3ec700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.634+0000 7f097d3ec700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:49.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.634+0000 7f097d3ec700 1 -- 172.21.15.40:0/1627497596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09781420b0 con 0x7f097813ac20 2023-09-30T00:52:49.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.850+0000 7f095effd700 1 -- 172.21.15.40:0/1627497596 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0970008e40 con 0x7f097813ac20 2023-09-30T00:52:49.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.850+0000 7f095effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:49.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.850+0000 7f09757fa700 4 set_mon_vals no callback set 2023-09-30T00:52:49.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.915+0000 7f095effd700 1 -- 172.21.15.40:0/1627497596 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0970025460 con 0x7f097813ac20 2023-09-30T00:52:49.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.915+0000 7f095effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:49.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.915+0000 7f095effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:52:49.916 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.915+0000 7f095effd700 10 monclient: dump: 2023-09-30T00:52:49.916 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:49.916 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:49.916 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:49.916 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:49.917 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:49.917 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:49.917 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:49.917 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:49.917 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:49.917 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:49.934 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.933+0000 7f095effd700 1 -- 172.21.15.40:0/1627497596 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0970003040 con 0x7f097813ac20 2023-09-30T00:52:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.934+0000 7f095effd700 1 --2- 172.21.15.40:0/1627497596 >> v2:172.21.15.135:6800/116426 conn(0x7f0964068890 0x7f096406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.934+0000 7f095effd700 1 -- 172.21.15.40:0/1627497596 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0970085bc0 con 0x7f097813ac20 2023-09-30T00:52:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.934+0000 7f095effd700 10 client.4764.objecter ms_dispatch 0x7f0978001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.934+0000 7f095effd700 3 client.4764.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.934+0000 7f095effd700 3 client.4764.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.934+0000 7f0976ffd700 1 --2- 172.21.15.40:0/1627497596 >> v2:172.21.15.135:6800/116426 conn(0x7f0964068890 0x7f096406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.934+0000 7f095effd700 20 client.4764.objecter dump_active .. 0 homeless 2023-09-30T00:52:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.934+0000 7f0976ffd700 10 monclient: get_auth_request con 0x7f0964068890 auth_method 0 2023-09-30T00:52:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.934+0000 7f095effd700 1 -- 172.21.15.40:0/1627497596 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f09700b0640 con 0x7f097813ac20 2023-09-30T00:52:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.934+0000 7f095effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:49.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.934+0000 7f095effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:49.936 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.934+0000 7f0976ffd700 1 --2- 172.21.15.40:0/1627497596 >> v2:172.21.15.135:6800/116426 conn(0x7f0964068890 0x7f096406ad40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0968009db0 tx=0x7f09680091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:49.936 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:49.934+0000 7f095effd700 10 client.4764.objecter ms_handle_connect 0x7f0964068890 2023-09-30T00:52:50.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.202+0000 7f097d3ec700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:50.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.202+0000 7f097d3ec700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:50.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.202+0000 7f097d3ec700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:52:50.203 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.202+0000 7f097d3ec700 1 -- 172.21.15.40:0/1627497596 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0978064a00 con 0x7f097813ac20 2023-09-30T00:52:50.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.207+0000 7f095effd700 1 -- 172.21.15.40:0/1627497596 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0970053470 con 0x7f097813ac20 2023-09-30T00:52:50.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.207+0000 7f095effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:50.208 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.207+0000 7f095effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:50.208 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:50.208 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:50.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.209+0000 7f097d3ec700 10 librados: watch_flush enter 2023-09-30T00:52:50.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 10 librados: watch_flush exit 2023-09-30T00:52:50.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 20 client.4764.objecter shutdown clearing up homeless session... 2023-09-30T00:52:50.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 10 client.4764.objecter successfully canceled tick 2023-09-30T00:52:50.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 1 -- 172.21.15.40:0/1627497596 >> v2:172.21.15.135:6800/116426 conn(0x7f0964068890 msgr2=0x7f096406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:50.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 1 --2- 172.21.15.40:0/1627497596 >> v2:172.21.15.135:6800/116426 conn(0x7f0964068890 0x7f096406ad40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0968009db0 tx=0x7f09680091a0 comp rx=0 tx=0).stop 2023-09-30T00:52:50.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 10 monclient: shutdown 2023-09-30T00:52:50.211 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:50.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 1 -- 172.21.15.40:0/1627497596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f097813ac20 msgr2=0x7f09780a7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:50.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 1 --2- 172.21.15.40:0/1627497596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f097813ac20 0x7f09780a7f30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f097000b610 tx=0x7f0970018000 comp rx=0 tx=0).stop 2023-09-30T00:52:50.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 1 -- 172.21.15.40:0/1627497596 shutdown_connections 2023-09-30T00:52:50.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 1 --2- 172.21.15.40:0/1627497596 >> v2:172.21.15.135:6800/116426 conn(0x7f0964068890 0x7f096406ad40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:50.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 1 --2- 172.21.15.40:0/1627497596 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f09780a5fb0 0x7f097813e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:50.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 1 --2- 172.21.15.40:0/1627497596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f097813ac20 0x7f09780a7f30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:50.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 1 --2- 172.21.15.40:0/1627497596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f09780b1460 0x7f09780a79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:50.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 1 -- 172.21.15.40:0/1627497596 >> 172.21.15.40:0/1627497596 conn(0x7f09780631f0 msgr2=0x7f09780afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:50.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 1 -- 172.21.15.40:0/1627497596 shutdown_connections 2023-09-30T00:52:50.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 1 -- 172.21.15.40:0/1627497596 wait complete. 2023-09-30T00:52:50.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:50.210+0000 7f097d3ec700 1 librados: shutdown 2023-09-30T00:52:53.225 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:53.363 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.363+0000 7f671ec23700 10 monclient: get_monmap_and_config 2023-09-30T00:52:53.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.363+0000 7f671ec23700 10 monclient: build_initial_monmap 2023-09-30T00:52:53.364 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.363+0000 7f671ec23700 10 monclient: monmap: 2023-09-30T00:52:53.364 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:53.364 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:53.364 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:53.364361+0000 2023-09-30T00:52:53.364 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:53.364361+0000 2023-09-30T00:52:53.364 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:53.364 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:53.364 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:53.365 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:53.365 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:53.365 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:53.365 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:53.365 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 1 Processor -- start 2023-09-30T00:52:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 1 -- start start 2023-09-30T00:52:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 10 monclient: init 2023-09-30T00:52:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-09-30T00:52:53.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67180b1460 0x7f67180b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 10 monclient(hunting): picked mon.noname-b con 0x7f67180b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f67180a5fb0 0x7f6718141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 10 monclient(hunting): picked mon.noname-c con 0x7f67180a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:52:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67180af410 addr v1:172.21.15.135:6789/0 2023-09-30T00:52:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67180a71f0 con 0x7f67180af410 2023-09-30T00:52:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67180a74d0 con 0x7f67180a5fb0 2023-09-30T00:52:53.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67180a63c0 con 0x7f67180b1460 2023-09-30T00:52:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.364+0000 7f671ec23700 10 monclient(hunting): authenticate will time out at 5634.665973s 2023-09-30T00:52:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.365+0000 7f671c9bf700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67180b1460 0x7f67180b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.366+0000 7f671c9bf700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67180b1460 0x7f67180b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49898/0 (socket says 172.21.15.40:49898) 2023-09-30T00:52:53.369 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.366+0000 7f671c9bf700 1 -- 172.21.15.40:0/1390974090 learned_addr learned my addr 172.21.15.40:0/1390974090 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.366+0000 7f671c9bf700 10 monclient(hunting): get_auth_request con 0x7f67180b1460 auth_method 0 2023-09-30T00:52:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.366+0000 7f671c9bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.366+0000 7f671c9bf700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.366+0000 7f671c9bf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.366+0000 7f6717fff700 1 --2- 172.21.15.40:0/1390974090 >> v2:172.21.15.135:3300/0 conn(0x7f67180a5fb0 0x7f6718141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.366+0000 7f671c9bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.366+0000 7f671c9bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.366+0000 7f671c9bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.366+0000 7f6717fff700 10 monclient(hunting): get_auth_request con 0x7f67180a5fb0 auth_method 0 2023-09-30T00:52:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.366+0000 7f6717fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.366+0000 7f6717fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:53.370 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.366+0000 7f6717fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.367+0000 7f6717fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.367+0000 7f6717fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:53.371 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.367+0000 7f6717fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.369+0000 7f671c9bf700 10 monclient(hunting): handle_auth_done global_id 4664 payload 274 2023-09-30T00:52:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.369+0000 7f671c9bf700 1 -- 172.21.15.40:0/1390974090 >> v2:172.21.15.135:3300/0 conn(0x7f67180a5fb0 msgr2=0x7f6718141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:52:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.369+0000 7f671c9bf700 1 --2- 172.21.15.40:0/1390974090 >> v2:172.21.15.135:3300/0 conn(0x7f67180a5fb0 0x7f6718141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.369+0000 7f671c9bf700 1 -- 172.21.15.40:0/1390974090 >> v1:172.21.15.135:6789/0 conn(0x7f67180af410 legacy=0x7f67180af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.369+0000 7f671c9bf700 10 monclient: _finish_hunting 0 2023-09-30T00:52:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.369+0000 7f671c9bf700 1 monclient: found mon.noname-b 2023-09-30T00:52:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.369+0000 7f671c9bf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.369+0000 7f671c9bf700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:52:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.369+0000 7f671c9bf700 1 -- 172.21.15.40:0/1390974090 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67180a56d0 con 0x7f67180b1460 2023-09-30T00:52:53.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.371+0000 7f671c9bf700 1 --2- 172.21.15.40:0/1390974090 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67180b1460 0x7f67180b1830 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f67080099b0 tx=0x7f6708009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb37e871da1ee7e5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f67177fe700 1 -- 172.21.15.40:0/1390974090 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6708009ea0 con 0x7f67180b1460 2023-09-30T00:52:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f67177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f67177fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:52:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f67177fe700 10 monclient: dump: 2023-09-30T00:52:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:53.376 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:53.377 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:53.377 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:53.377 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:53.377 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:53.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f67177fe700 10 monclient: _finish_auth 0 2023-09-30T00:52:53.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f67177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:53.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f67177fe700 1 -- 172.21.15.40:0/1390974090 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670800b860 con 0x7f67180b1460 2023-09-30T00:52:53.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f671ec23700 5 monclient: authenticate success, global_id 4664 2023-09-30T00:52:53.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f67177fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:53.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f671ec23700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:53.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f67177fe700 1 -- 172.21.15.40:0/1390974090 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670800bc00 con 0x7f67180b1460 2023-09-30T00:52:53.377 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f671ec23700 4 set_mon_vals no callback set 2023-09-30T00:52:53.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f67177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:53.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f67177fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:52:53.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.375+0000 7f67177fe700 10 monclient: dump: 2023-09-30T00:52:53.378 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:53.378 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:53.378 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:53.378 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:53.378 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:53.378 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:53.378 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:53.378 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:53.378 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:53.379 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.376+0000 7f671ec23700 10 monclient: shutdown 2023-09-30T00:52:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.376+0000 7f671ec23700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.376+0000 7f671ec23700 1 -- 172.21.15.40:0/1390974090 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67180b1460 msgr2=0x7f67180b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.376+0000 7f671ec23700 1 --2- 172.21.15.40:0/1390974090 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67180b1460 0x7f67180b1830 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f67080099b0 tx=0x7f6708009250 comp rx=0 tx=0).stop 2023-09-30T00:52:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.376+0000 7f671ec23700 1 -- 172.21.15.40:0/1390974090 shutdown_connections 2023-09-30T00:52:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.376+0000 7f671ec23700 1 --2- 172.21.15.40:0/1390974090 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67180b1460 0x7f67180b1830 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.376+0000 7f671ec23700 1 --2- 172.21.15.40:0/1390974090 >> v2:172.21.15.135:3300/0 conn(0x7f67180a5fb0 0x7f6718141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.376+0000 7f671ec23700 1 -- 172.21.15.40:0/1390974090 >> 172.21.15.40:0/1390974090 conn(0x7f67180631f0 msgr2=0x7f67180635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.376+0000 7f671ec23700 1 -- 172.21.15.40:0/1390974090 shutdown_connections 2023-09-30T00:52:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.376+0000 7f671ec23700 1 -- 172.21.15.40:0/1390974090 wait complete. 2023-09-30T00:52:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.377+0000 7f671ec23700 10 monclient: build_initial_monmap 2023-09-30T00:52:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.377+0000 7f671ec23700 10 monclient: monmap: 2023-09-30T00:52:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:53.382 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:53.383 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.377+0000 7f671ec23700 1 librados: starting msgr at 2023-09-30T00:52:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.377+0000 7f671ec23700 1 librados: starting objecter 2023-09-30T00:52:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.377+0000 7f671ec23700 1 Processor -- start 2023-09-30T00:52:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.377+0000 7f671ec23700 1 -- start start 2023-09-30T00:52:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.377+0000 7f671ec23700 1 librados: setting wanted keys 2023-09-30T00:52:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.377+0000 7f671ec23700 1 librados: calling monclient init 2023-09-30T00:52:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.377+0000 7f671ec23700 10 monclient: init 2023-09-30T00:52:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:52:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67180a5fb0 0x7f67180a9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 10 monclient(hunting): picked mon.noname-a con 0x7f67180a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67180af410 0x7f671813e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 10 monclient(hunting): picked mon.noname-b con 0x7f67180af410 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67180b1460 0x7f67180a9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 10 monclient(hunting): picked mon.noname-c con 0x7f67180b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67180aad50 con 0x7f67180a5fb0 2023-09-30T00:52:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671813ef70 con 0x7f67180af410 2023-09-30T00:52:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67180ae590 con 0x7f67180b1460 2023-09-30T00:52:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.378+0000 7f671ec23700 10 monclient(hunting): authenticate will time out at 5634.679465s 2023-09-30T00:52:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f671d1c0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67180b1460 0x7f67180a9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f671c9bf700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67180a5fb0 0x7f67180a9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f6717fff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67180af410 0x7f671813e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f671d1c0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67180b1460 0x7f67180a9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:49906/0 (socket says 172.21.15.40:49906) 2023-09-30T00:52:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f671d1c0700 1 -- 172.21.15.40:0/3246641751 learned_addr learned my addr 172.21.15.40:0/3246641751 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f671d1c0700 10 monclient(hunting): get_auth_request con 0x7f67180b1460 auth_method 0 2023-09-30T00:52:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f671d1c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f671d1c0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:53.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f671d1c0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f6717fff700 10 monclient(hunting): get_auth_request con 0x7f67180af410 auth_method 0 2023-09-30T00:52:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f6717fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f6717fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f6717fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f671c9bf700 10 monclient(hunting): get_auth_request con 0x7f67180a5fb0 auth_method 0 2023-09-30T00:52:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f671c9bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f671c9bf700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.379+0000 7f671c9bf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.380+0000 7f671d1c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.380+0000 7f671d1c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.380+0000 7f671d1c0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:53.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.380+0000 7f6717fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:53.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.380+0000 7f6717fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:53.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.380+0000 7f6717fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:53.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.380+0000 7f671c9bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:53.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.380+0000 7f671c9bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:53.387 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.380+0000 7f671c9bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:53.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.383+0000 7f671d1c0700 10 monclient(hunting): handle_auth_done global_id 4667 payload 824 2023-09-30T00:52:53.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.383+0000 7f671d1c0700 1 -- 172.21.15.40:0/3246641751 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67180af410 msgr2=0x7f671813e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:53.388 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.383+0000 7f671d1c0700 1 --2- 172.21.15.40:0/3246641751 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67180af410 0x7f671813e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.383+0000 7f671d1c0700 1 -- 172.21.15.40:0/3246641751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67180a5fb0 msgr2=0x7f67180a9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.383+0000 7f671d1c0700 1 --2- 172.21.15.40:0/3246641751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67180a5fb0 0x7f67180a9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.383+0000 7f671d1c0700 10 monclient: _finish_hunting 0 2023-09-30T00:52:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.383+0000 7f671d1c0700 1 monclient: found mon.noname-c 2023-09-30T00:52:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.383+0000 7f671d1c0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.383+0000 7f671d1c0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:52:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.383+0000 7f671d1c0700 1 -- 172.21.15.40:0/3246641751 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67080094f0 con 0x7f67180b1460 2023-09-30T00:52:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.386+0000 7f671d1c0700 1 --2- 172.21.15.40:0/3246641751 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67180b1460 0x7f67180a9f60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f671000c8e0 tx=0x7f6710001710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.386+0000 7f6714ff9700 10 client.?.objecter ms_handle_connect 0x7f67180b1460 2023-09-30T00:52:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.386+0000 7f6714ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:53.391 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.390+0000 7f6714ff9700 1 -- 172.21.15.40:0/3246641751 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710012630 con 0x7f67180b1460 2023-09-30T00:52:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.390+0000 7f6714ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.390+0000 7f6714ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:52:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.390+0000 7f6714ff9700 10 monclient: dump: 2023-09-30T00:52:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:53.393 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.390+0000 7f6714ff9700 10 monclient: _finish_auth 0 2023-09-30T00:52:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.390+0000 7f6714ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f6714ff9700 1 -- 172.21.15.40:0/3246641751 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6710012790 con 0x7f67180b1460 2023-09-30T00:52:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f6714ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f6714ff9700 1 -- 172.21.15.40:0/3246641751 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710012b50 con 0x7f67180b1460 2023-09-30T00:52:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f671ec23700 5 monclient: authenticate success, global_id 4667 2023-09-30T00:52:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f671ec23700 10 monclient: _renew_subs 2023-09-30T00:52:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f671ec23700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f671ec23700 1 -- 172.21.15.40:0/3246641751 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6718151200 con 0x7f67180b1460 2023-09-30T00:52:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f6716ffd700 4 set_mon_vals no callback set 2023-09-30T00:52:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f6714ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f6714ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:52:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f6714ff9700 10 monclient: dump: 2023-09-30T00:52:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:53.394 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f671ec23700 10 client.4667.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f671ec23700 10 monclient: _renew_subs 2023-09-30T00:52:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f671ec23700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f671ec23700 1 -- 172.21.15.40:0/3246641751 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67180aa6e0 con 0x7f67180b1460 2023-09-30T00:52:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.391+0000 7f671ec23700 1 librados: init done 2023-09-30T00:52:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.392+0000 7f671ec23700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:53.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.392+0000 7f671ec23700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:53.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.392+0000 7f671ec23700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:53.396 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.392+0000 7f671ec23700 1 -- 172.21.15.40:0/3246641751 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671813e670 con 0x7f67180b1460 2023-09-30T00:52:53.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.647+0000 7f6714ff9700 1 -- 172.21.15.40:0/3246641751 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6710024420 con 0x7f67180b1460 2023-09-30T00:52:53.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.647+0000 7f6714ff9700 1 --2- 172.21.15.40:0/3246641751 >> v2:172.21.15.135:6800/116426 conn(0x7f67040689e0 0x7f670406ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:53.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.647+0000 7f6714ff9700 1 -- 172.21.15.40:0/3246641751 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6710084790 con 0x7f67180b1460 2023-09-30T00:52:53.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.647+0000 7f6714ff9700 10 client.4667.objecter ms_dispatch 0x7f6718001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:53.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.647+0000 7f6714ff9700 3 client.4667.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:53.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.647+0000 7f6714ff9700 3 client.4667.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:53.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.648+0000 7f6714ff9700 20 client.4667.objecter dump_active .. 0 homeless 2023-09-30T00:52:53.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.648+0000 7f671c9bf700 1 --2- 172.21.15.40:0/3246641751 >> v2:172.21.15.135:6800/116426 conn(0x7f67040689e0 0x7f670406ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:53.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.648+0000 7f671c9bf700 10 monclient: get_auth_request con 0x7f67040689e0 auth_method 0 2023-09-30T00:52:53.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.648+0000 7f671c9bf700 1 --2- 172.21.15.40:0/3246641751 >> v2:172.21.15.135:6800/116426 conn(0x7f67040689e0 0x7f670406ae90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f67080099b0 tx=0x7f6708017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:53.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.648+0000 7f6714ff9700 10 client.4667.objecter ms_handle_connect 0x7f67040689e0 2023-09-30T00:52:53.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.689+0000 7f6714ff9700 1 -- 172.21.15.40:0/3246641751 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f67100520b0 con 0x7f67180b1460 2023-09-30T00:52:53.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.689+0000 7f6714ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:53.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.689+0000 7f6714ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:53.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.946+0000 7f671ec23700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:53.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.946+0000 7f671ec23700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:53.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.946+0000 7f671ec23700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:52:53.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.946+0000 7f671ec23700 1 -- 172.21.15.40:0/3246641751 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f67180a9c80 con 0x7f67180b1460 2023-09-30T00:52:53.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.951+0000 7f6714ff9700 1 -- 172.21.15.40:0/3246641751 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f67180a9c80 con 0x7f67180b1460 2023-09-30T00:52:53.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.951+0000 7f6714ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:53.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.951+0000 7f6714ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:53.952 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:53.952 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:53.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 10 librados: watch_flush enter 2023-09-30T00:52:53.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 10 librados: watch_flush exit 2023-09-30T00:52:53.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 20 client.4667.objecter shutdown clearing up homeless session... 2023-09-30T00:52:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 10 client.4667.objecter successfully canceled tick 2023-09-30T00:52:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 1 -- 172.21.15.40:0/3246641751 >> v2:172.21.15.135:6800/116426 conn(0x7f67040689e0 msgr2=0x7f670406ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 1 --2- 172.21.15.40:0/3246641751 >> v2:172.21.15.135:6800/116426 conn(0x7f67040689e0 0x7f670406ae90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f67080099b0 tx=0x7f6708017000 comp rx=0 tx=0).stop 2023-09-30T00:52:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 10 monclient: shutdown 2023-09-30T00:52:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 1 -- 172.21.15.40:0/3246641751 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67180b1460 msgr2=0x7f67180a9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 1 --2- 172.21.15.40:0/3246641751 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67180b1460 0x7f67180a9f60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f671000c8e0 tx=0x7f6710001710 comp rx=0 tx=0).stop 2023-09-30T00:52:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 1 -- 172.21.15.40:0/3246641751 shutdown_connections 2023-09-30T00:52:53.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 1 --2- 172.21.15.40:0/3246641751 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f67180b1460 0x7f67180a9f60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 1 --2- 172.21.15.40:0/3246641751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f67180a5fb0 0x7f67180a9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 1 --2- 172.21.15.40:0/3246641751 >> v2:172.21.15.135:6800/116426 conn(0x7f67040689e0 0x7f670406ae90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 1 --2- 172.21.15.40:0/3246641751 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f67180af410 0x7f671813e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 1 -- 172.21.15.40:0/3246641751 >> 172.21.15.40:0/3246641751 conn(0x7f67180631f0 msgr2=0x7f6718144aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 1 -- 172.21.15.40:0/3246641751 shutdown_connections 2023-09-30T00:52:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 1 -- 172.21.15.40:0/3246641751 wait complete. 2023-09-30T00:52:53.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:53.954+0000 7f671ec23700 1 librados: shutdown 2023-09-30T00:52:54.283 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:06:29.994 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:54.289 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:06:30.001 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:54.317 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:06:30.028 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:54.340 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:06:30.027 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:54.342 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:06:30.029 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:54.347 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:06:30.059 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:54.350 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:06:30.062 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:54.364 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:06:30.051 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:54.375 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:06:30.087 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:52:56.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:52:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.108+0000 7fa24f15d700 10 monclient: get_monmap_and_config 2023-09-30T00:52:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.108+0000 7fa24f15d700 10 monclient: build_initial_monmap 2023-09-30T00:52:57.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.108+0000 7fa24f15d700 10 monclient: monmap: 2023-09-30T00:52:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:52:57.109566+0000 2023-09-30T00:52:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:52:57.109566+0000 2023-09-30T00:52:57.110 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:57.111 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:57.111 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:52:57.111 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:52:57.111 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:52:57.111 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:52:57.111 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:52:57.112 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.109+0000 7fa24f15d700 1 Processor -- start 2023-09-30T00:52:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.109+0000 7fa24f15d700 1 -- start start 2023-09-30T00:52:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.109+0000 7fa24f15d700 10 monclient: init 2023-09-30T00:52:57.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.109+0000 7fa24f15d700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.109+0000 7fa24f15d700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-09-30T00:52:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.110+0000 7fa24f15d700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa2480b1460 0x7fa2480b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.110+0000 7fa24f15d700 10 monclient(hunting): picked mon.noname-c con 0x7fa2480b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:52:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.110+0000 7fa24f15d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa2480a5fb0 addr v1:172.21.15.40:6789/0 2023-09-30T00:52:57.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.110+0000 7fa24f15d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa24813ac20 addr v1:172.21.15.135:6789/0 2023-09-30T00:52:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.110+0000 7fa24f15d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2480a5b20 con 0x7fa2480a5fb0 2023-09-30T00:52:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.110+0000 7fa24f15d700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2480a71f0 con 0x7fa24813ac20 2023-09-30T00:52:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.110+0000 7fa24f15d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.110+0000 7fa24f15d700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2480a74d0 con 0x7fa2480b1460 2023-09-30T00:52:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.110+0000 7fa24f15d700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:57.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.110+0000 7fa24f15d700 10 monclient(hunting): authenticate will time out at 5638.411084s 2023-09-30T00:52:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.110+0000 7fa247fff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa2480a5fb0 0x7fa248141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41518/0 (socket says 172.21.15.40:41518) 2023-09-30T00:52:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.110+0000 7fa247fff700 1 -- 172.21.15.40:0/3805805763 learned_addr learned my addr 172.21.15.40:0/3805805763 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:52:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.110+0000 7fa24cef9700 1 --2- 172.21.15.40:0/3805805763 >> v2:172.21.15.135:3300/0 conn(0x7fa2480b1460 0x7fa2480b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.111+0000 7fa24cef9700 10 monclient(hunting): get_auth_request con 0x7fa2480b1460 auth_method 0 2023-09-30T00:52:57.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.111+0000 7fa24cef9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.111+0000 7fa24cef9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.111+0000 7fa24cef9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.112+0000 7fa24cef9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.112+0000 7fa24cef9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:57.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.112+0000 7fa24cef9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.114+0000 7fa2477fe700 1 -- 172.21.15.40:0/3805805763 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914640947 0 0) 0x7fa2480a5b20 con 0x7fa2480a5fb0 2023-09-30T00:52:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.114+0000 7fa2477fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.114+0000 7fa2477fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.114+0000 7fa2477fe700 10 monclient(hunting): my global_id is 4770 2023-09-30T00:52:57.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.114+0000 7fa2477fe700 1 -- 172.21.15.40:0/3805805763 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa238003570 con 0x7fa2480a5fb0 2023-09-30T00:52:57.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.115+0000 7fa24cef9700 10 monclient(hunting): handle_auth_done global_id 4846 payload 274 2023-09-30T00:52:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.115+0000 7fa24cef9700 1 -- 172.21.15.40:0/3805805763 >> v1:172.21.15.135:6789/0 conn(0x7fa24813ac20 legacy=0x7fa24813b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.116+0000 7fa24cef9700 1 -- 172.21.15.40:0/3805805763 >> v1:172.21.15.40:6789/0 conn(0x7fa2480a5fb0 legacy=0x7fa248141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.116+0000 7fa24cef9700 10 monclient: _finish_hunting 0 2023-09-30T00:52:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.116+0000 7fa24cef9700 1 monclient: found mon.noname-c 2023-09-30T00:52:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.116+0000 7fa24cef9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:57.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.116+0000 7fa24cef9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:52:57.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.116+0000 7fa24cef9700 1 -- 172.21.15.40:0/3805805763 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2480a63c0 con 0x7fa2480b1460 2023-09-30T00:52:57.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.118+0000 7fa24cef9700 1 --2- 172.21.15.40:0/3805805763 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2480b1460 0x7fa2480b1830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa23c009910 tx=0x7fa23c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=829244d317822f78 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:57.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa2477fe700 1 -- 172.21.15.40:0/3805805763 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c00b640 con 0x7fa2480b1460 2023-09-30T00:52:57.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa2477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:57.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa2477fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:52:57.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa2477fe700 10 monclient: dump: 2023-09-30T00:52:57.127 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:57.128 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:57.128 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:57.128 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:57.128 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:57.128 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:57.129 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:57.129 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:57.129 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:57.129 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:57.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa2477fe700 10 monclient: _finish_auth 0 2023-09-30T00:52:57.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa2477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:57.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa2477fe700 1 -- 172.21.15.40:0/3805805763 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23c00b7a0 con 0x7fa2480b1460 2023-09-30T00:52:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa2477fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa2477fe700 1 -- 172.21.15.40:0/3805805763 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c00bb60 con 0x7fa2480b1460 2023-09-30T00:52:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa2477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa2477fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:52:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa2477fe700 10 monclient: dump: 2023-09-30T00:52:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:57.133 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:57.134 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa24f15d700 5 monclient: authenticate success, global_id 4846 2023-09-30T00:52:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa24f15d700 10 monclient: get_monmap_and_config success 2023-09-30T00:52:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.126+0000 7fa24f15d700 4 set_mon_vals no callback set 2023-09-30T00:52:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.127+0000 7fa24f15d700 10 monclient: shutdown 2023-09-30T00:52:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.127+0000 7fa24f15d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.127+0000 7fa24f15d700 1 -- 172.21.15.40:0/3805805763 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2480b1460 msgr2=0x7fa2480b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.127+0000 7fa24f15d700 1 --2- 172.21.15.40:0/3805805763 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2480b1460 0x7fa2480b1830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa23c009910 tx=0x7fa23c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:52:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.127+0000 7fa24f15d700 1 -- 172.21.15.40:0/3805805763 shutdown_connections 2023-09-30T00:52:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.127+0000 7fa24f15d700 1 --2- 172.21.15.40:0/3805805763 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2480b1460 0x7fa2480b1830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.127+0000 7fa24f15d700 1 -- 172.21.15.40:0/3805805763 >> 172.21.15.40:0/3805805763 conn(0x7fa2480631f0 msgr2=0x7fa2480635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.127+0000 7fa24f15d700 1 -- 172.21.15.40:0/3805805763 shutdown_connections 2023-09-30T00:52:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.127+0000 7fa24f15d700 1 -- 172.21.15.40:0/3805805763 wait complete. 2023-09-30T00:52:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.128+0000 7fa24f15d700 10 monclient: build_initial_monmap 2023-09-30T00:52:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.128+0000 7fa24f15d700 10 monclient: monmap: 2023-09-30T00:52:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:52:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:52:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:52:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:52:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:52:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:52:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:52:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:52:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:52:57.136 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.128+0000 7fa24f15d700 1 librados: starting msgr at 2023-09-30T00:52:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.128+0000 7fa24f15d700 1 librados: starting objecter 2023-09-30T00:52:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.128+0000 7fa24f15d700 1 Processor -- start 2023-09-30T00:52:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.128+0000 7fa24f15d700 1 -- start start 2023-09-30T00:52:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.128+0000 7fa24f15d700 1 librados: setting wanted keys 2023-09-30T00:52:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.128+0000 7fa24f15d700 1 librados: calling monclient init 2023-09-30T00:52:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.128+0000 7fa24f15d700 10 monclient: init 2023-09-30T00:52:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 10 monclient: _reopen_session rank -1 2023-09-30T00:52:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:52:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2480a5fb0 0x7fa24813e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 10 monclient(hunting): picked mon.noname-b con 0x7fa2480a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:52:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2480b1460 0x7fa2480aa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 10 monclient(hunting): picked mon.noname-c con 0x7fa2480b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:52:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa24813ac20 0x7fa2480a79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 10 monclient(hunting): picked mon.noname-a con 0x7fa24813ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:52:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2480aaae0 con 0x7fa24813ac20 2023-09-30T00:52:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2480abc80 con 0x7fa2480a5fb0 2023-09-30T00:52:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:52:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2480aa080 con 0x7fa2480b1460 2023-09-30T00:52:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 10 monclient(hunting): _renew_subs 2023-09-30T00:52:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24f15d700 10 monclient(hunting): authenticate will time out at 5638.430438s 2023-09-30T00:52:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.129+0000 7fa24cef9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2480a5fb0 0x7fa24813e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.130+0000 7fa24d6fa700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa24813ac20 0x7fa2480a79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.130+0000 7fa24cef9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2480a5fb0 0x7fa24813e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:39462/0 (socket says 172.21.15.40:39462) 2023-09-30T00:52:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.130+0000 7fa24cef9700 1 -- 172.21.15.40:0/1494789069 learned_addr learned my addr 172.21.15.40:0/1494789069 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:52:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.130+0000 7fa24cef9700 10 monclient(hunting): get_auth_request con 0x7fa2480a5fb0 auth_method 0 2023-09-30T00:52:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.130+0000 7fa24cef9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.130+0000 7fa24cef9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.130+0000 7fa24cef9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:57.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.130+0000 7fa24d6fa700 10 monclient(hunting): get_auth_request con 0x7fa24813ac20 auth_method 0 2023-09-30T00:52:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.130+0000 7fa24d6fa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:52:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.130+0000 7fa24d6fa700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:52:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.130+0000 7fa24d6fa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:52:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.130+0000 7fa24cef9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.130+0000 7fa24cef9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.130+0000 7fa24cef9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.131+0000 7fa24d6fa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:52:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.131+0000 7fa24d6fa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:52:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.131+0000 7fa24d6fa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:52:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.134+0000 7fa24cef9700 10 monclient(hunting): handle_auth_done global_id 4852 payload 824 2023-09-30T00:52:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.134+0000 7fa24cef9700 1 -- 172.21.15.40:0/1494789069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2480b1460 msgr2=0x7fa2480aa5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.134+0000 7fa24cef9700 1 --2- 172.21.15.40:0/1494789069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2480b1460 0x7fa2480aa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.134+0000 7fa24cef9700 1 -- 172.21.15.40:0/1494789069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa24813ac20 msgr2=0x7fa2480a79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.134+0000 7fa24cef9700 1 --2- 172.21.15.40:0/1494789069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa24813ac20 0x7fa2480a79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.134+0000 7fa24cef9700 10 monclient: _finish_hunting 0 2023-09-30T00:52:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.134+0000 7fa24cef9700 1 monclient: found mon.noname-b 2023-09-30T00:52:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.134+0000 7fa24cef9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:52:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.134+0000 7fa24cef9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:52:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.134+0000 7fa24cef9700 1 -- 172.21.15.40:0/1494789069 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23c009420 con 0x7fa2480a5fb0 2023-09-30T00:52:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.137+0000 7fa24cef9700 1 --2- 172.21.15.40:0/1494789069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2480a5fb0 0x7fa24813e700 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa23c009fb0 tx=0x7fa23c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.137+0000 7fa244ff9700 10 client.?.objecter ms_handle_connect 0x7fa2480a5fb0 2023-09-30T00:52:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.137+0000 7fa244ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:52:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa244ff9700 1 -- 172.21.15.40:0/1494789069 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c01f070 con 0x7fa2480a5fb0 2023-09-30T00:52:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa244ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa244ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:52:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa244ff9700 10 monclient: dump: 2023-09-30T00:52:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:57.143 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa244ff9700 10 monclient: _finish_auth 0 2023-09-30T00:52:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa244ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:52:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa244ff9700 1 -- 172.21.15.40:0/1494789069 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa23c00b910 con 0x7fa2480a5fb0 2023-09-30T00:52:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa244ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:52:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa244ff9700 1 -- 172.21.15.40:0/1494789069 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa23c011420 con 0x7fa2480a5fb0 2023-09-30T00:52:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa244ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:52:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa244ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:52:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa244ff9700 10 monclient: dump: 2023-09-30T00:52:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:52:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:52:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:52:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:52:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:52:57.144 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:52:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:52:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:52:57.145 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:52:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa24f15d700 5 monclient: authenticate success, global_id 4852 2023-09-30T00:52:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa24f15d700 10 monclient: _renew_subs 2023-09-30T00:52:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa24f15d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa24f15d700 1 -- 172.21.15.40:0/1494789069 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2480a7f30 con 0x7fa2480a5fb0 2023-09-30T00:52:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa246ffd700 4 set_mon_vals no callback set 2023-09-30T00:52:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa24f15d700 10 client.4852.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:52:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa24f15d700 10 monclient: _renew_subs 2023-09-30T00:52:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa24f15d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.141+0000 7fa24f15d700 1 -- 172.21.15.40:0/1494789069 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2480a9860 con 0x7fa2480a5fb0 2023-09-30T00:52:57.145 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.142+0000 7fa24f15d700 1 librados: init done 2023-09-30T00:52:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.142+0000 7fa24f15d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.143+0000 7fa24f15d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.143+0000 7fa24f15d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.143+0000 7fa24f15d700 1 -- 172.21.15.40:0/1494789069 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa248064a00 con 0x7fa2480a5fb0 2023-09-30T00:52:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.162+0000 7fa244ff9700 1 -- 172.21.15.40:0/1494789069 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa23c0117b0 con 0x7fa2480a5fb0 2023-09-30T00:52:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.162+0000 7fa244ff9700 1 --2- 172.21.15.40:0/1494789069 >> v2:172.21.15.135:6800/116426 conn(0x7fa22c068890 0x7fa22c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:52:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.163+0000 7fa247fff700 1 --2- 172.21.15.40:0/1494789069 >> v2:172.21.15.135:6800/116426 conn(0x7fa22c068890 0x7fa22c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:52:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.163+0000 7fa247fff700 10 monclient: get_auth_request con 0x7fa22c068890 auth_method 0 2023-09-30T00:52:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.163+0000 7fa244ff9700 10 client.4852.objecter ms_handle_connect 0x7fa22c068890 2023-09-30T00:52:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.163+0000 7fa247fff700 1 --2- 172.21.15.40:0/1494789069 >> v2:172.21.15.135:6800/116426 conn(0x7fa22c068890 0x7fa22c06ad40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa2340058d0 tx=0x7fa2340079d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:52:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.398+0000 7fa244ff9700 1 -- 172.21.15.40:0/1494789069 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa23c057520 con 0x7fa2480a5fb0 2023-09-30T00:52:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.398+0000 7fa244ff9700 10 client.4852.objecter ms_dispatch 0x7fa248001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:52:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.398+0000 7fa244ff9700 3 client.4852.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:52:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.398+0000 7fa244ff9700 3 client.4852.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:52:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.398+0000 7fa244ff9700 20 client.4852.objecter dump_active .. 0 homeless 2023-09-30T00:52:57.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.408+0000 7fa244ff9700 1 -- 172.21.15.40:0/1494789069 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa23c053160 con 0x7fa2480a5fb0 2023-09-30T00:52:57.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.408+0000 7fa244ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:52:57.409 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.408+0000 7fa244ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:52:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.671+0000 7fa24f15d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:52:57.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.671+0000 7fa24f15d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.671+0000 7fa24f15d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:52:57.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.671+0000 7fa24f15d700 1 -- 172.21.15.40:0/1494789069 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa2480a56d0 con 0x7fa2480a5fb0 2023-09-30T00:52:57.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.677+0000 7fa244ff9700 1 -- 172.21.15.40:0/1494789069 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa23c056780 con 0x7fa2480a5fb0 2023-09-30T00:52:57.677 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.677+0000 7fa244ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:52:57.678 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.677+0000 7fa244ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:52:57.678 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:52:57.678 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:52:57.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.679+0000 7fa24f15d700 10 librados: watch_flush enter 2023-09-30T00:52:57.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.679+0000 7fa24f15d700 10 librados: watch_flush exit 2023-09-30T00:52:57.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.679+0000 7fa24f15d700 20 client.4852.objecter shutdown clearing up homeless session... 2023-09-30T00:52:57.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.679+0000 7fa24f15d700 10 client.4852.objecter successfully canceled tick 2023-09-30T00:52:57.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.679+0000 7fa24f15d700 1 -- 172.21.15.40:0/1494789069 >> v2:172.21.15.135:6800/116426 conn(0x7fa22c068890 msgr2=0x7fa22c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:57.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.679+0000 7fa24f15d700 1 --2- 172.21.15.40:0/1494789069 >> v2:172.21.15.135:6800/116426 conn(0x7fa22c068890 0x7fa22c06ad40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa2340058d0 tx=0x7fa2340079d0 comp rx=0 tx=0).stop 2023-09-30T00:52:57.681 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.679+0000 7fa24f15d700 10 monclient: shutdown 2023-09-30T00:52:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.679+0000 7fa24f15d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:52:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.679+0000 7fa24f15d700 1 -- 172.21.15.40:0/1494789069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2480a5fb0 msgr2=0x7fa24813e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:52:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.679+0000 7fa24f15d700 1 --2- 172.21.15.40:0/1494789069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2480a5fb0 0x7fa24813e700 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa23c009fb0 tx=0x7fa23c017000 comp rx=0 tx=0).stop 2023-09-30T00:52:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.680+0000 7fa24f15d700 1 -- 172.21.15.40:0/1494789069 shutdown_connections 2023-09-30T00:52:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.680+0000 7fa24f15d700 1 --2- 172.21.15.40:0/1494789069 >> v2:172.21.15.135:6800/116426 conn(0x7fa22c068890 0x7fa22c06ad40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.680+0000 7fa24f15d700 1 --2- 172.21.15.40:0/1494789069 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa2480a5fb0 0x7fa24813e700 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.680+0000 7fa24f15d700 1 --2- 172.21.15.40:0/1494789069 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa24813ac20 0x7fa2480a79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.680+0000 7fa24f15d700 1 --2- 172.21.15.40:0/1494789069 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa2480b1460 0x7fa2480aa5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:52:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.680+0000 7fa24f15d700 1 -- 172.21.15.40:0/1494789069 >> 172.21.15.40:0/1494789069 conn(0x7fa2480631f0 msgr2=0x7fa2480b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:52:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.680+0000 7fa24f15d700 1 -- 172.21.15.40:0/1494789069 shutdown_connections 2023-09-30T00:52:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.680+0000 7fa24f15d700 1 -- 172.21.15.40:0/1494789069 wait complete. 2023-09-30T00:52:57.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:52:57.680+0000 7fa24f15d700 1 librados: shutdown 2023-09-30T00:53:00.695 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:53:00.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.837+0000 7f959fd57700 10 monclient: get_monmap_and_config 2023-09-30T00:53:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.837+0000 7f959fd57700 10 monclient: build_initial_monmap 2023-09-30T00:53:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.837+0000 7f959fd57700 10 monclient: monmap: 2023-09-30T00:53:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:00.838344+0000 2023-09-30T00:53:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:00.838344+0000 2023-09-30T00:53:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:00.838 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:00.839 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:00.839 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.837+0000 7f959fd57700 1 Processor -- start 2023-09-30T00:53:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 1 -- start start 2023-09-30T00:53:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 10 monclient: init 2023-09-30T00:53:00.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-09-30T00:53:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95980b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:53:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f95980b1830 0x7f95980a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 10 monclient(hunting): picked mon.noname-c con 0x7f95980b1830 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f959813ac20 0x7f959813b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 10 monclient(hunting): picked mon.noname-a con 0x7f959813ac20 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95980a5b20 con 0x7f95980b1460 2023-09-30T00:53:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9598141e40 con 0x7f959813ac20 2023-09-30T00:53:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95980a64f0 con 0x7f95980b1830 2023-09-30T00:53:00.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:00.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.838+0000 7f959fd57700 10 monclient(hunting): authenticate will time out at 5642.139966s 2023-09-30T00:53:00.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.839+0000 7f959e2f4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f959813ac20 0x7f959813b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:00.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.840+0000 7f959d2f2700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f95980b1830 0x7f95980a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:00.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.840+0000 7f959e2f4700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f959813ac20 0x7f959813b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44580/0 (socket says 172.21.15.40:44580) 2023-09-30T00:53:00.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.840+0000 7f959e2f4700 1 -- 172.21.15.40:0/25212709 learned_addr learned my addr 172.21.15.40:0/25212709 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:00.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.840+0000 7f959e2f4700 10 monclient(hunting): get_auth_request con 0x7f959813ac20 auth_method 0 2023-09-30T00:53:00.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.840+0000 7f959e2f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:00.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.840+0000 7f959e2f4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:00.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.840+0000 7f959e2f4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:00.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.840+0000 7f959d2f2700 10 monclient(hunting): get_auth_request con 0x7f95980b1830 auth_method 0 2023-09-30T00:53:00.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.840+0000 7f959d2f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:00.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.840+0000 7f959d2f2700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:00.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.840+0000 7f959d2f2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:00.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.841+0000 7f959e2f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:00.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.841+0000 7f959e2f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:00.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.841+0000 7f959e2f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:00.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.841+0000 7f959d2f2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:00.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.841+0000 7f959d2f2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:00.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.841+0000 7f959d2f2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:00.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.844+0000 7f959e2f4700 10 monclient(hunting): handle_auth_done global_id 4776 payload 274 2023-09-30T00:53:00.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.844+0000 7f959e2f4700 1 -- 172.21.15.40:0/25212709 >> v2:172.21.15.135:3300/0 conn(0x7f95980b1830 msgr2=0x7f95980a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:00.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.844+0000 7f959e2f4700 1 --2- 172.21.15.40:0/25212709 >> v2:172.21.15.135:3300/0 conn(0x7f95980b1830 0x7f95980a5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:00.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.844+0000 7f959e2f4700 1 -- 172.21.15.40:0/25212709 >> v1:172.21.15.40:6789/0 conn(0x7f95980b1460 legacy=0x7f9598141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:00.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.844+0000 7f959e2f4700 10 monclient: _finish_hunting 0 2023-09-30T00:53:00.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.844+0000 7f959e2f4700 1 monclient: found mon.noname-a 2023-09-30T00:53:00.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.844+0000 7f959e2f4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:00.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.844+0000 7f959e2f4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:53:00.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.844+0000 7f959e2f4700 1 -- 172.21.15.40:0/25212709 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95980a71f0 con 0x7f959813ac20 2023-09-30T00:53:00.850 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.849+0000 7f959e2f4700 1 --2- 172.21.15.40:0/25212709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f959813ac20 0x7f959813b080 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9590009f50 tx=0x7f9590009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f21f3c57f8ec43ee server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959caf1700 1 -- 172.21.15.40:0/25212709 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959000e070 con 0x7f959813ac20 2023-09-30T00:53:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959caf1700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959caf1700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:53:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959caf1700 10 monclient: dump: 2023-09-30T00:53:00.856 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:00.857 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:00.857 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:00.857 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:00.857 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:00.857 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:00.857 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:00.857 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:00.857 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:00.857 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:00.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959caf1700 10 monclient: _finish_auth 0 2023-09-30T00:53:00.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959caf1700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:00.857 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959caf1700 1 -- 172.21.15.40:0/25212709 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959000b980 con 0x7f959813ac20 2023-09-30T00:53:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959fd57700 5 monclient: authenticate success, global_id 4776 2023-09-30T00:53:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959fd57700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959caf1700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959fd57700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959caf1700 1 -- 172.21.15.40:0/25212709 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95900092e0 con 0x7f959813ac20 2023-09-30T00:53:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959caf1700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959fd57700 4 set_mon_vals no callback set 2023-09-30T00:53:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959caf1700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:53:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.854+0000 7f959caf1700 10 monclient: dump: 2023-09-30T00:53:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:00.858 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:00.859 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.855+0000 7f959fd57700 10 monclient: shutdown 2023-09-30T00:53:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.855+0000 7f959fd57700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.855+0000 7f959fd57700 1 -- 172.21.15.40:0/25212709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f959813ac20 msgr2=0x7f959813b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.855+0000 7f959fd57700 1 --2- 172.21.15.40:0/25212709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f959813ac20 0x7f959813b080 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9590009f50 tx=0x7f9590009840 comp rx=0 tx=0).stop 2023-09-30T00:53:00.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.855+0000 7f959fd57700 1 -- 172.21.15.40:0/25212709 shutdown_connections 2023-09-30T00:53:00.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.855+0000 7f959fd57700 1 --2- 172.21.15.40:0/25212709 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f959813ac20 0x7f959813b080 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:00.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.855+0000 7f959fd57700 1 --2- 172.21.15.40:0/25212709 >> v2:172.21.15.135:3300/0 conn(0x7f95980b1830 0x7f95980a5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:00.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.855+0000 7f959fd57700 1 -- 172.21.15.40:0/25212709 >> 172.21.15.40:0/25212709 conn(0x7f95980631f0 msgr2=0x7f95980635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:00.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.855+0000 7f959fd57700 1 -- 172.21.15.40:0/25212709 shutdown_connections 2023-09-30T00:53:00.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.855+0000 7f959fd57700 1 -- 172.21.15.40:0/25212709 wait complete. 2023-09-30T00:53:00.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.856+0000 7f959fd57700 10 monclient: build_initial_monmap 2023-09-30T00:53:00.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.856+0000 7f959fd57700 10 monclient: monmap: 2023-09-30T00:53:00.862 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:00.862 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:00.863 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.857+0000 7f959fd57700 1 librados: starting msgr at 2023-09-30T00:53:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.857+0000 7f959fd57700 1 librados: starting objecter 2023-09-30T00:53:00.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.857+0000 7f959fd57700 1 Processor -- start 2023-09-30T00:53:00.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.857+0000 7f959fd57700 1 -- start start 2023-09-30T00:53:00.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.857+0000 7f959fd57700 1 librados: setting wanted keys 2023-09-30T00:53:00.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.857+0000 7f959fd57700 1 librados: calling monclient init 2023-09-30T00:53:00.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.857+0000 7f959fd57700 10 monclient: init 2023-09-30T00:53:00.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:00.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:53:00.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95980b1460 0x7f9598150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:00.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 10 monclient(hunting): picked mon.noname-b con 0x7f95980b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:00.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95980b1830 0x7f959813d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:00.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 10 monclient(hunting): picked mon.noname-a con 0x7f95980b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:00.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f959813ac20 0x7f959813da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:00.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 10 monclient(hunting): picked mon.noname-c con 0x7f959813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9598143b00 con 0x7f95980b1830 2023-09-30T00:53:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959813d030 con 0x7f95980b1460 2023-09-30T00:53:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959814b070 con 0x7f959813ac20 2023-09-30T00:53:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959fd57700 10 monclient(hunting): authenticate will time out at 5642.159148s 2023-09-30T00:53:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959d2f2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95980b1830 0x7f959813d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.858+0000 7f959daf3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95980b1460 0x7f9598150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.859+0000 7f959d2f2700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95980b1830 0x7f959813d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44594/0 (socket says 172.21.15.40:44594) 2023-09-30T00:53:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.859+0000 7f959d2f2700 1 -- 172.21.15.40:0/1232060077 learned_addr learned my addr 172.21.15.40:0/1232060077 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.859+0000 7f959d2f2700 10 monclient(hunting): get_auth_request con 0x7f95980b1830 auth_method 0 2023-09-30T00:53:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.859+0000 7f959d2f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.859+0000 7f959d2f2700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.859+0000 7f959d2f2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.859+0000 7f959daf3700 10 monclient(hunting): get_auth_request con 0x7f95980b1460 auth_method 0 2023-09-30T00:53:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.859+0000 7f959daf3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.859+0000 7f959daf3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.859+0000 7f959daf3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.859+0000 7f959d2f2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.859+0000 7f959d2f2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.859+0000 7f959d2f2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.860+0000 7f959daf3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:00.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.860+0000 7f959daf3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:00.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.860+0000 7f959daf3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:00.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.860+0000 7f959e2f4700 1 --2- 172.21.15.40:0/1232060077 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f959813ac20 0x7f959813da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:00.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.861+0000 7f959e2f4700 10 monclient(hunting): get_auth_request con 0x7f959813ac20 auth_method 0 2023-09-30T00:53:00.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.861+0000 7f959e2f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:00.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.861+0000 7f959e2f4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:00.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.861+0000 7f959e2f4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:00.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.862+0000 7f959e2f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:00.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.862+0000 7f959e2f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:00.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.862+0000 7f959e2f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.863+0000 7f959d2f2700 10 monclient(hunting): handle_auth_done global_id 4779 payload 824 2023-09-30T00:53:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.863+0000 7f959d2f2700 1 -- 172.21.15.40:0/1232060077 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f959813ac20 msgr2=0x7f959813da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.863+0000 7f959d2f2700 1 --2- 172.21.15.40:0/1232060077 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f959813ac20 0x7f959813da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.863+0000 7f959d2f2700 1 -- 172.21.15.40:0/1232060077 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95980b1460 msgr2=0x7f9598150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:00.868 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.863+0000 7f959d2f2700 1 --2- 172.21.15.40:0/1232060077 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95980b1460 0x7f9598150ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.863+0000 7f959d2f2700 10 monclient: _finish_hunting 0 2023-09-30T00:53:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.863+0000 7f959d2f2700 1 monclient: found mon.noname-a 2023-09-30T00:53:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.863+0000 7f959d2f2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.863+0000 7f959d2f2700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:53:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.863+0000 7f959d2f2700 1 -- 172.21.15.40:0/1232060077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9590009ae0 con 0x7f95980b1830 2023-09-30T00:53:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.866+0000 7f959d2f2700 1 --2- 172.21.15.40:0/1232060077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95980b1830 0x7f959813d4c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f958800ecb0 tx=0x7f95880091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.866+0000 7f958dffb700 10 client.?.objecter ms_handle_connect 0x7f95980b1830 2023-09-30T00:53:00.869 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.866+0000 7f958dffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f958dffb700 1 -- 172.21.15.40:0/1232060077 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958800c850 con 0x7f95980b1830 2023-09-30T00:53:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f958dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f958dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:53:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f958dffb700 10 monclient: dump: 2023-09-30T00:53:00.871 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:00.872 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f958dffb700 10 monclient: _finish_auth 0 2023-09-30T00:53:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f958dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:00.872 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f958dffb700 1 -- 172.21.15.40:0/1232060077 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958800c9b0 con 0x7f95980b1830 2023-09-30T00:53:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f958dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f958dffb700 1 -- 172.21.15.40:0/1232060077 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958800cd50 con 0x7f95980b1830 2023-09-30T00:53:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f959fd57700 5 monclient: authenticate success, global_id 4779 2023-09-30T00:53:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f959fd57700 10 monclient: _renew_subs 2023-09-30T00:53:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f959fd57700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:53:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f959fd57700 1 -- 172.21.15.40:0/1232060077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f959813df40 con 0x7f95980b1830 2023-09-30T00:53:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f958ffff700 4 set_mon_vals no callback set 2023-09-30T00:53:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f958dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f958dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:53:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.870+0000 7f958dffb700 10 monclient: dump: 2023-09-30T00:53:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:00.874 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.871+0000 7f959fd57700 10 client.4779.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.871+0000 7f959fd57700 10 monclient: _renew_subs 2023-09-30T00:53:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.871+0000 7f959fd57700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:53:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.871+0000 7f959fd57700 1 -- 172.21.15.40:0/1232060077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95981518b0 con 0x7f95980b1830 2023-09-30T00:53:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.871+0000 7f959fd57700 1 librados: init done 2023-09-30T00:53:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.872+0000 7f959fd57700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.872+0000 7f959fd57700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.872+0000 7f959fd57700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:53:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.872+0000 7f959fd57700 1 -- 172.21.15.40:0/1232060077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9598064a00 con 0x7f95980b1830 2023-09-30T00:53:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.899+0000 7f958dffb700 1 -- 172.21.15.40:0/1232060077 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9588013680 con 0x7f95980b1830 2023-09-30T00:53:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.899+0000 7f958dffb700 1 --2- 172.21.15.40:0/1232060077 >> v2:172.21.15.135:6800/116426 conn(0x7f957c068890 0x7f957c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.899+0000 7f958dffb700 1 -- 172.21.15.40:0/1232060077 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9588084880 con 0x7f95980b1830 2023-09-30T00:53:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.899+0000 7f958dffb700 10 client.4779.objecter ms_dispatch 0x7f9598001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.899+0000 7f958dffb700 3 client.4779.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.899+0000 7f958dffb700 3 client.4779.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.899+0000 7f958dffb700 20 client.4779.objecter dump_active .. 0 homeless 2023-09-30T00:53:00.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.899+0000 7f959daf3700 1 --2- 172.21.15.40:0/1232060077 >> v2:172.21.15.135:6800/116426 conn(0x7f957c068890 0x7f957c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.899+0000 7f959daf3700 10 monclient: get_auth_request con 0x7f957c068890 auth_method 0 2023-09-30T00:53:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.900+0000 7f959daf3700 1 --2- 172.21.15.40:0/1232060077 >> v2:172.21.15.135:6800/116426 conn(0x7f957c068890 0x7f957c06ad40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9594003aa0 tx=0x7f9594000bc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:00.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:00.900+0000 7f958dffb700 10 client.4779.objecter ms_handle_connect 0x7f957c068890 2023-09-30T00:53:01.178 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.178+0000 7f958dffb700 1 -- 172.21.15.40:0/1232060077 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f95880521a0 con 0x7f95980b1830 2023-09-30T00:53:01.178 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.178+0000 7f958dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:01.178 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.178+0000 7f958dffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.434+0000 7f959fd57700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:53:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.434+0000 7f959fd57700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:53:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.434+0000 7f959fd57700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:53:01.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.434+0000 7f959fd57700 1 -- 172.21.15.40:0/1232060077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f959806f9b0 con 0x7f95980b1830 2023-09-30T00:53:01.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.440+0000 7f958dffb700 1 -- 172.21.15.40:0/1232060077 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f95880557c0 con 0x7f95980b1830 2023-09-30T00:53:01.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.440+0000 7f958dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:53:01.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.440+0000 7f958dffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:53:01.441 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:53:01.441 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:53:01.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 10 librados: watch_flush enter 2023-09-30T00:53:01.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 10 librados: watch_flush exit 2023-09-30T00:53:01.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 20 client.4779.objecter shutdown clearing up homeless session... 2023-09-30T00:53:01.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 10 client.4779.objecter successfully canceled tick 2023-09-30T00:53:01.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 1 -- 172.21.15.40:0/1232060077 >> v2:172.21.15.135:6800/116426 conn(0x7f957c068890 msgr2=0x7f957c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:01.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 1 --2- 172.21.15.40:0/1232060077 >> v2:172.21.15.135:6800/116426 conn(0x7f957c068890 0x7f957c06ad40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9594003aa0 tx=0x7f9594000bc0 comp rx=0 tx=0).stop 2023-09-30T00:53:01.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 10 monclient: shutdown 2023-09-30T00:53:01.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:01.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 1 -- 172.21.15.40:0/1232060077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95980b1830 msgr2=0x7f959813d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:01.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 1 --2- 172.21.15.40:0/1232060077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95980b1830 0x7f959813d4c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f958800ecb0 tx=0x7f95880091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:01.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 1 -- 172.21.15.40:0/1232060077 shutdown_connections 2023-09-30T00:53:01.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 1 --2- 172.21.15.40:0/1232060077 >> v2:172.21.15.135:6800/116426 conn(0x7f957c068890 0x7f957c06ad40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:01.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 1 --2- 172.21.15.40:0/1232060077 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f95980b1460 0x7f9598150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:01.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 1 --2- 172.21.15.40:0/1232060077 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f959813ac20 0x7f959813da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:01.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 1 --2- 172.21.15.40:0/1232060077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f95980b1830 0x7f959813d4c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:01.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 1 -- 172.21.15.40:0/1232060077 >> 172.21.15.40:0/1232060077 conn(0x7f95980631f0 msgr2=0x7f95980af310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:01.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 1 -- 172.21.15.40:0/1232060077 shutdown_connections 2023-09-30T00:53:01.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 1 -- 172.21.15.40:0/1232060077 wait complete. 2023-09-30T00:53:01.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:01.442+0000 7f959fd57700 1 librados: shutdown 2023-09-30T00:53:04.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:53:04.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.591+0000 7f15cd7e4700 10 monclient: get_monmap_and_config 2023-09-30T00:53:04.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.591+0000 7f15cd7e4700 10 monclient: build_initial_monmap 2023-09-30T00:53:04.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.591+0000 7f15cd7e4700 10 monclient: monmap: 2023-09-30T00:53:04.592 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:04.592 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:04.592 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:04.592514+0000 2023-09-30T00:53:04.592 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:04.592514+0000 2023-09-30T00:53:04.592 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:04.592 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:04.592 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:04.593 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:04.593 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:04.593 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:04.593 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:04.593 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 1 Processor -- start 2023-09-30T00:53:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 1 -- start start 2023-09-30T00:53:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 10 monclient: init 2023-09-30T00:53:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-09-30T00:53:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f15c80b16a0 0x7f15c80a7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 10 monclient(hunting): picked mon.noname-c con 0x7f15c80b16a0 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:04.595 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15c80a54c0 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15c813aeb0 0x7f15c813b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 10 monclient(hunting): picked mon.noname-b con 0x7f15c813aeb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c80a7f00 con 0x7f15c80a54c0 2023-09-30T00:53:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15c80a8390 con 0x7f15c80b16a0 2023-09-30T00:53:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c80a6070 con 0x7f15c813aeb0 2023-09-30T00:53:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.592+0000 7f15cd7e4700 10 monclient(hunting): authenticate will time out at 5645.893857s 2023-09-30T00:53:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.593+0000 7f15c77fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15c813aeb0 0x7f15c813b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:04.596 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.593+0000 7f15c6ffd700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f15c80b16a0 0x7f15c80a7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.593+0000 7f15c77fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15c813aeb0 0x7f15c813b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:54874/0 (socket says 172.21.15.40:54874) 2023-09-30T00:53:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.593+0000 7f15c77fe700 1 -- 172.21.15.40:0/2530764250 learned_addr learned my addr 172.21.15.40:0/2530764250 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.593+0000 7f15c77fe700 10 monclient(hunting): get_auth_request con 0x7f15c813aeb0 auth_method 0 2023-09-30T00:53:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.593+0000 7f15c77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.593+0000 7f15c77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.593+0000 7f15c77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.594+0000 7f15c6ffd700 10 monclient(hunting): get_auth_request con 0x7f15c80b16a0 auth_method 0 2023-09-30T00:53:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.594+0000 7f15c6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.594+0000 7f15c6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.594+0000 7f15c6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.594+0000 7f15c77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:04.597 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.594+0000 7f15c77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.594+0000 7f15c77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.595+0000 7f15c6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.595+0000 7f15c6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.595+0000 7f15c6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:04.598 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.596+0000 7f15c77fe700 10 monclient(hunting): handle_auth_done global_id 4673 payload 274 2023-09-30T00:53:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.596+0000 7f15c77fe700 1 -- 172.21.15.40:0/2530764250 >> v2:172.21.15.135:3300/0 conn(0x7f15c80b16a0 msgr2=0x7f15c80a7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.596+0000 7f15c77fe700 1 --2- 172.21.15.40:0/2530764250 >> v2:172.21.15.135:3300/0 conn(0x7f15c80b16a0 0x7f15c80a7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.596+0000 7f15c77fe700 1 -- 172.21.15.40:0/2530764250 >> v1:172.21.15.135:6789/0 conn(0x7f15c80a54c0 legacy=0x7f15c80a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.596+0000 7f15c77fe700 10 monclient: _finish_hunting 0 2023-09-30T00:53:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.596+0000 7f15c77fe700 1 monclient: found mon.noname-b 2023-09-30T00:53:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.596+0000 7f15c77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.596+0000 7f15c77fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:53:04.599 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.596+0000 7f15c77fe700 1 -- 172.21.15.40:0/2530764250 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c80b1a70 con 0x7f15c813aeb0 2023-09-30T00:53:04.600 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.599+0000 7f15c77fe700 1 --2- 172.21.15.40:0/2530764250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15c813aeb0 0x7f15c813b310 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009f20 tx=0x7f15c0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba269e36122229be server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:04.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.602+0000 7f15c5ffb700 1 -- 172.21.15.40:0/2530764250 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c000e070 con 0x7f15c813aeb0 2023-09-30T00:53:04.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.602+0000 7f15c5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:04.602 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.602+0000 7f15c5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:04.603 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.602+0000 7f15c5ffb700 10 monclient: dump: 2023-09-30T00:53:04.603 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:04.603 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:04.603 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:04.603 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:04.603 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:04.603 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:04.603 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:04.603 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:04.603 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:04.603 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.602+0000 7f15c5ffb700 10 monclient: _finish_auth 0 2023-09-30T00:53:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.602+0000 7f15c5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.602+0000 7f15cd7e4700 5 monclient: authenticate success, global_id 4673 2023-09-30T00:53:04.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.602+0000 7f15cd7e4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:04.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15c5ffb700 1 -- 172.21.15.40:0/2530764250 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c000b9d0 con 0x7f15c813aeb0 2023-09-30T00:53:04.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15c5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:04.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15c5ffb700 1 -- 172.21.15.40:0/2530764250 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c00092e0 con 0x7f15c813aeb0 2023-09-30T00:53:04.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15cd7e4700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:04.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15cd7e4700 4 set_mon_vals no callback set 2023-09-30T00:53:04.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15c5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:04.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15c5ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:53:04.605 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15c5ffb700 10 monclient: dump: 2023-09-30T00:53:04.606 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:04.606 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:04.606 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:04.606 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:04.606 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:04.606 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:04.606 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:04.606 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:04.606 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:04.606 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:04.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15cd7e4700 10 monclient: shutdown 2023-09-30T00:53:04.606 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15cd7e4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15cd7e4700 1 -- 172.21.15.40:0/2530764250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15c813aeb0 msgr2=0x7f15c813b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15cd7e4700 1 --2- 172.21.15.40:0/2530764250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15c813aeb0 0x7f15c813b310 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009f20 tx=0x7f15c0009840 comp rx=0 tx=0).stop 2023-09-30T00:53:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15cd7e4700 1 -- 172.21.15.40:0/2530764250 shutdown_connections 2023-09-30T00:53:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15cd7e4700 1 --2- 172.21.15.40:0/2530764250 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15c813aeb0 0x7f15c813b310 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15cd7e4700 1 --2- 172.21.15.40:0/2530764250 >> v2:172.21.15.135:3300/0 conn(0x7f15c80b16a0 0x7f15c80a7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15cd7e4700 1 -- 172.21.15.40:0/2530764250 >> 172.21.15.40:0/2530764250 conn(0x7f15c806f130 msgr2=0x7f15c80631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15cd7e4700 1 -- 172.21.15.40:0/2530764250 shutdown_connections 2023-09-30T00:53:04.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.604+0000 7f15cd7e4700 1 -- 172.21.15.40:0/2530764250 wait complete. 2023-09-30T00:53:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.605+0000 7f15cd7e4700 10 monclient: build_initial_monmap 2023-09-30T00:53:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.605+0000 7f15cd7e4700 10 monclient: monmap: 2023-09-30T00:53:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:04.610 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:04.611 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.606+0000 7f15cd7e4700 1 librados: starting msgr at 2023-09-30T00:53:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.606+0000 7f15cd7e4700 1 librados: starting objecter 2023-09-30T00:53:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.606+0000 7f15cd7e4700 1 Processor -- start 2023-09-30T00:53:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.606+0000 7f15cd7e4700 1 -- start start 2023-09-30T00:53:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.606+0000 7f15cd7e4700 1 librados: setting wanted keys 2023-09-30T00:53:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.606+0000 7f15cd7e4700 1 librados: calling monclient init 2023-09-30T00:53:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.606+0000 7f15cd7e4700 10 monclient: init 2023-09-30T00:53:04.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.606+0000 7f15cd7e4700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:53:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15c80a54c0 0x7f15c80aa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 10 monclient(hunting): picked mon.noname-a con 0x7f15c80a54c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15c80b16a0 0x7f15c814dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 10 monclient(hunting): picked mon.noname-b con 0x7f15c80b16a0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15c813aeb0 0x7f15c813c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 10 monclient(hunting): picked mon.noname-c con 0x7f15c813aeb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c813da80 con 0x7f15c80a54c0 2023-09-30T00:53:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c813bf50 con 0x7f15c80b16a0 2023-09-30T00:53:04.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:04.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c8140060 con 0x7f15c813aeb0 2023-09-30T00:53:04.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:04.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15cd7e4700 10 monclient(hunting): authenticate will time out at 5645.908103s 2023-09-30T00:53:04.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15c77fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15c813aeb0 0x7f15c813c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:04.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15c6ffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15c80a54c0 0x7f15c80aa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:04.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.607+0000 7f15c67fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15c80b16a0 0x7f15c814dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:04.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c77fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15c813aeb0 0x7f15c813c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:54880/0 (socket says 172.21.15.40:54880) 2023-09-30T00:53:04.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c77fe700 1 -- 172.21.15.40:0/4219522762 learned_addr learned my addr 172.21.15.40:0/4219522762 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:04.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c77fe700 10 monclient(hunting): get_auth_request con 0x7f15c813aeb0 auth_method 0 2023-09-30T00:53:04.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:04.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:04.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:04.613 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c67fc700 10 monclient(hunting): get_auth_request con 0x7f15c80b16a0 auth_method 0 2023-09-30T00:53:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c67fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c6ffd700 10 monclient(hunting): get_auth_request con 0x7f15c80a54c0 auth_method 0 2023-09-30T00:53:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:04.614 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.608+0000 7f15c67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.609+0000 7f15c6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.609+0000 7f15c6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:04.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.609+0000 7f15c6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.612+0000 7f15c77fe700 10 monclient(hunting): handle_auth_done global_id 4676 payload 824 2023-09-30T00:53:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.612+0000 7f15c77fe700 1 -- 172.21.15.40:0/4219522762 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15c80b16a0 msgr2=0x7f15c814dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.612+0000 7f15c77fe700 1 --2- 172.21.15.40:0/4219522762 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15c80b16a0 0x7f15c814dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.612+0000 7f15c77fe700 1 -- 172.21.15.40:0/4219522762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15c80a54c0 msgr2=0x7f15c80aa8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.612+0000 7f15c77fe700 1 --2- 172.21.15.40:0/4219522762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15c80a54c0 0x7f15c80aa8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.612+0000 7f15c77fe700 10 monclient: _finish_hunting 0 2023-09-30T00:53:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.612+0000 7f15c77fe700 1 monclient: found mon.noname-c 2023-09-30T00:53:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.612+0000 7f15c77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.612+0000 7f15c77fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:53:04.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.612+0000 7f15c77fe700 1 -- 172.21.15.40:0/4219522762 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c0009ae0 con 0x7f15c813aeb0 2023-09-30T00:53:04.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.612+0000 7f15c67fc700 1 --2- 172.21.15.40:0/4219522762 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15c80b16a0 0x7f15c814dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:53:04.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.615+0000 7f15c77fe700 1 --2- 172.21.15.40:0/4219522762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15c813aeb0 0x7f15c813c3e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009430 tx=0x7f15c0018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:04.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.615+0000 7f15aeffd700 10 client.?.objecter ms_handle_connect 0x7f15c813aeb0 2023-09-30T00:53:04.617 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.615+0000 7f15aeffd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15aeffd700 1 -- 172.21.15.40:0/4219522762 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c0007af0 con 0x7f15c813aeb0 2023-09-30T00:53:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:04.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15aeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:53:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15aeffd700 10 monclient: dump: 2023-09-30T00:53:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:04.621 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15aeffd700 10 monclient: _finish_auth 0 2023-09-30T00:53:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15aeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15aeffd700 1 -- 172.21.15.40:0/4219522762 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c0007c90 con 0x7f15c813aeb0 2023-09-30T00:53:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15aeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15aeffd700 1 -- 172.21.15.40:0/4219522762 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c001e680 con 0x7f15c813aeb0 2023-09-30T00:53:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15aeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:53:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15aeffd700 10 monclient: dump: 2023-09-30T00:53:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:04.623 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15cd7e4700 5 monclient: authenticate success, global_id 4676 2023-09-30T00:53:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15cd7e4700 10 monclient: _renew_subs 2023-09-30T00:53:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15cd7e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:53:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15cd7e4700 1 -- 172.21.15.40:0/4219522762 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c814dff0 con 0x7f15c813aeb0 2023-09-30T00:53:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15c57fa700 4 set_mon_vals no callback set 2023-09-30T00:53:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15cd7e4700 10 client.4676.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15cd7e4700 10 monclient: _renew_subs 2023-09-30T00:53:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15cd7e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:53:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15cd7e4700 1 -- 172.21.15.40:0/4219522762 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c813cb60 con 0x7f15c813aeb0 2023-09-30T00:53:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.619+0000 7f15cd7e4700 1 librados: init done 2023-09-30T00:53:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.620+0000 7f15cd7e4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.620+0000 7f15cd7e4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.620+0000 7f15cd7e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:53:04.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.620+0000 7f15cd7e4700 1 -- 172.21.15.40:0/4219522762 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c813c0e0 con 0x7f15c813aeb0 2023-09-30T00:53:04.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.900+0000 7f15aeffd700 1 -- 172.21.15.40:0/4219522762 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f15c0003040 con 0x7f15c813aeb0 2023-09-30T00:53:04.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.900+0000 7f15aeffd700 1 --2- 172.21.15.40:0/4219522762 >> v2:172.21.15.135:6800/116426 conn(0x7f15b4068960 0x7f15b406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:04.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.900+0000 7f15aeffd700 1 -- 172.21.15.40:0/4219522762 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f15c0086120 con 0x7f15c813aeb0 2023-09-30T00:53:04.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.900+0000 7f15aeffd700 10 client.4676.objecter ms_dispatch 0x7f15c8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:04.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.900+0000 7f15aeffd700 3 client.4676.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:04.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.900+0000 7f15aeffd700 3 client.4676.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:04.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.900+0000 7f15aeffd700 20 client.4676.objecter dump_active .. 0 homeless 2023-09-30T00:53:04.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.900+0000 7f15c6ffd700 1 --2- 172.21.15.40:0/4219522762 >> v2:172.21.15.135:6800/116426 conn(0x7f15b4068960 0x7f15b406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:04.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.901+0000 7f15c6ffd700 10 monclient: get_auth_request con 0x7f15b4068960 auth_method 0 2023-09-30T00:53:04.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.901+0000 7f15c6ffd700 1 --2- 172.21.15.40:0/4219522762 >> v2:172.21.15.135:6800/116426 conn(0x7f15b4068960 0x7f15b406ae10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009db0 tx=0x7f15b80091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:04.902 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.901+0000 7f15aeffd700 10 client.4676.objecter ms_handle_connect 0x7f15b4068960 2023-09-30T00:53:04.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.911+0000 7f15aeffd700 1 -- 172.21.15.40:0/4219522762 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f15c0053a40 con 0x7f15c813aeb0 2023-09-30T00:53:04.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.911+0000 7f15aeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:04.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:04.911+0000 7f15aeffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.168+0000 7f15cd7e4700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:53:05.169 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.169+0000 7f15cd7e4700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:53:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.169+0000 7f15cd7e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:53:05.170 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.169+0000 7f15cd7e4700 1 -- 172.21.15.40:0/4219522762 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f15c814e1e0 con 0x7f15c813aeb0 2023-09-30T00:53:05.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.261+0000 7f15aeffd700 1 -- 172.21.15.40:0/4219522762 <== mon.2 v2:172.21.15.40:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f15c0057060 con 0x7f15c813aeb0 2023-09-30T00:53:05.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.261+0000 7f15aeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:53:05.262 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.261+0000 7f15aeffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:53:05.262 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:53:05.262 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:53:05.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.263+0000 7f15cd7e4700 10 librados: watch_flush enter 2023-09-30T00:53:05.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 10 librados: watch_flush exit 2023-09-30T00:53:05.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 20 client.4676.objecter shutdown clearing up homeless session... 2023-09-30T00:53:05.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 10 client.4676.objecter successfully canceled tick 2023-09-30T00:53:05.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 1 -- 172.21.15.40:0/4219522762 >> v2:172.21.15.135:6800/116426 conn(0x7f15b4068960 msgr2=0x7f15b406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:05.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 1 --2- 172.21.15.40:0/4219522762 >> v2:172.21.15.135:6800/116426 conn(0x7f15b4068960 0x7f15b406ae10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f15b8009db0 tx=0x7f15b80091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 10 monclient: shutdown 2023-09-30T00:53:05.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:05.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 1 -- 172.21.15.40:0/4219522762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15c813aeb0 msgr2=0x7f15c813c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:05.265 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 1 --2- 172.21.15.40:0/4219522762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15c813aeb0 0x7f15c813c3e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f15c0009430 tx=0x7f15c0018000 comp rx=0 tx=0).stop 2023-09-30T00:53:05.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 1 -- 172.21.15.40:0/4219522762 shutdown_connections 2023-09-30T00:53:05.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 1 --2- 172.21.15.40:0/4219522762 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f15c813aeb0 0x7f15c813c3e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 1 --2- 172.21.15.40:0/4219522762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f15c80a54c0 0x7f15c80aa8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 1 --2- 172.21.15.40:0/4219522762 >> v2:172.21.15.135:6800/116426 conn(0x7f15b4068960 0x7f15b406ae10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 1 --2- 172.21.15.40:0/4219522762 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f15c80b16a0 0x7f15c814dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 1 -- 172.21.15.40:0/4219522762 >> 172.21.15.40:0/4219522762 conn(0x7f15c806f130 msgr2=0x7f15c80afb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:05.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 1 -- 172.21.15.40:0/4219522762 shutdown_connections 2023-09-30T00:53:05.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 1 -- 172.21.15.40:0/4219522762 wait complete. 2023-09-30T00:53:05.266 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.264+0000 7f15cd7e4700 1 librados: shutdown 2023-09-30T00:53:05.278 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-30T00:53:05.278 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:53:05.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.412+0000 7f6a6d325700 10 monclient: get_monmap_and_config 2023-09-30T00:53:05.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.412+0000 7f6a6d325700 10 monclient: build_initial_monmap 2023-09-30T00:53:05.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.412+0000 7f6a6d325700 10 monclient: monmap: 2023-09-30T00:53:05.413 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:05.413 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:05.413 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:05.413649+0000 2023-09-30T00:53:05.413 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:05.413649+0000 2023-09-30T00:53:05.413 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:05.413 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:05.414 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:05.414 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:05.414 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:05.414 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:05.414 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:05.414 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 1 Processor -- start 2023-09-30T00:53:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 1 -- start start 2023-09-30T00:53:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 10 monclient: init 2023-09-30T00:53:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-09-30T00:53:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a680a5710 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6a6813e1d0 0x7f6a6813e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 10 monclient(hunting): picked mon.noname-c con 0x7f6a6813e1d0 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6813eb60 0x7f6a6813ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 10 monclient(hunting): picked mon.noname-b con 0x7f6a6813eb60 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a680a5ae0 con 0x7f6a680a5710 2023-09-30T00:53:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a680a7ff0 con 0x7f6a6813e1d0 2023-09-30T00:53:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680a7cf0 con 0x7f6a6813eb60 2023-09-30T00:53:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.413+0000 7f6a6d325700 10 monclient(hunting): authenticate will time out at 5646.714926s 2023-09-30T00:53:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.414+0000 7f6a677fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6813eb60 0x7f6a6813ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.414+0000 7f6a667fc700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f6a6813e1d0 0x7f6a6813e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.414+0000 7f6a677fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6813eb60 0x7f6a6813ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:54884/0 (socket says 172.21.15.40:54884) 2023-09-30T00:53:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.414+0000 7f6a677fe700 1 -- 172.21.15.40:0/2838744966 learned_addr learned my addr 172.21.15.40:0/2838744966 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.414+0000 7f6a677fe700 10 monclient(hunting): get_auth_request con 0x7f6a6813eb60 auth_method 0 2023-09-30T00:53:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.414+0000 7f6a677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.414+0000 7f6a677fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.414+0000 7f6a677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.415+0000 7f6a667fc700 10 monclient(hunting): get_auth_request con 0x7f6a6813e1d0 auth_method 0 2023-09-30T00:53:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.415+0000 7f6a667fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.415+0000 7f6a667fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.415+0000 7f6a667fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.415+0000 7f6a677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.415+0000 7f6a677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.415+0000 7f6a677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:05.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.416+0000 7f6a667fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:05.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.416+0000 7f6a667fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:05.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.416+0000 7f6a667fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:05.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.418+0000 7f6a677fe700 10 monclient(hunting): handle_auth_done global_id 4679 payload 274 2023-09-30T00:53:05.419 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.418+0000 7f6a677fe700 1 -- 172.21.15.40:0/2838744966 >> v2:172.21.15.135:3300/0 conn(0x7f6a6813e1d0 msgr2=0x7f6a6813e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.418+0000 7f6a677fe700 1 --2- 172.21.15.40:0/2838744966 >> v2:172.21.15.135:3300/0 conn(0x7f6a6813e1d0 0x7f6a6813e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.418+0000 7f6a677fe700 1 -- 172.21.15.40:0/2838744966 >> v1:172.21.15.135:6789/0 conn(0x7f6a680a5710 legacy=0x7f6a680a75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.418+0000 7f6a677fe700 10 monclient: _finish_hunting 0 2023-09-30T00:53:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.418+0000 7f6a677fe700 1 monclient: found mon.noname-b 2023-09-30T00:53:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.418+0000 7f6a677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.418+0000 7f6a677fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:53:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.418+0000 7f6a677fe700 1 -- 172.21.15.40:0/2838744966 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680aedc0 con 0x7f6a6813eb60 2023-09-30T00:53:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.420+0000 7f6a677fe700 1 --2- 172.21.15.40:0/2838744966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6813eb60 0x7f6a6813ae40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009f20 tx=0x7f6a60009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6280e9d070eab368 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:05.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a65ffb700 1 -- 172.21.15.40:0/2838744966 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6000e070 con 0x7f6a6813eb60 2023-09-30T00:53:05.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:05.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a65ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:05.426 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a65ffb700 10 monclient: dump: 2023-09-30T00:53:05.426 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:05.426 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:05.426 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:05.426 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:05.427 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:05.427 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:05.427 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:05.427 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:05.427 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:05.427 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:05.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a65ffb700 10 monclient: _finish_auth 0 2023-09-30T00:53:05.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a65ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:05.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a65ffb700 1 -- 172.21.15.40:0/2838744966 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6000ba20 con 0x7f6a6813eb60 2023-09-30T00:53:05.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a6d325700 5 monclient: authenticate success, global_id 4679 2023-09-30T00:53:05.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a6d325700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:05.427 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a65ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a6d325700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a6d325700 4 set_mon_vals no callback set 2023-09-30T00:53:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a65ffb700 1 -- 172.21.15.40:0/2838744966 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a600092e0 con 0x7f6a6813eb60 2023-09-30T00:53:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a65ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:53:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a65ffb700 10 monclient: dump: 2023-09-30T00:53:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:05.428 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:05.429 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a6d325700 10 monclient: shutdown 2023-09-30T00:53:05.429 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a6d325700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:05.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a6d325700 1 -- 172.21.15.40:0/2838744966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6813eb60 msgr2=0x7f6a6813ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:05.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a6d325700 1 --2- 172.21.15.40:0/2838744966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6813eb60 0x7f6a6813ae40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009f20 tx=0x7f6a60009840 comp rx=0 tx=0).stop 2023-09-30T00:53:05.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a6d325700 1 -- 172.21.15.40:0/2838744966 shutdown_connections 2023-09-30T00:53:05.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a6d325700 1 --2- 172.21.15.40:0/2838744966 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6813eb60 0x7f6a6813ae40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a6d325700 1 --2- 172.21.15.40:0/2838744966 >> v2:172.21.15.135:3300/0 conn(0x7f6a6813e1d0 0x7f6a6813e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a6d325700 1 -- 172.21.15.40:0/2838744966 >> 172.21.15.40:0/2838744966 conn(0x7f6a68069080 msgr2=0x7f6a680631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:05.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a6d325700 1 -- 172.21.15.40:0/2838744966 shutdown_connections 2023-09-30T00:53:05.430 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.425+0000 7f6a6d325700 1 -- 172.21.15.40:0/2838744966 wait complete. 2023-09-30T00:53:05.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.426+0000 7f6a6d325700 10 monclient: build_initial_monmap 2023-09-30T00:53:05.433 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.426+0000 7f6a6d325700 10 monclient: monmap: 2023-09-30T00:53:05.434 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:05.434 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:05.434 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:05.434 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:05.434 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:05.434 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:05.434 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:05.434 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:05.434 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:05.434 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:05.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 1 librados: starting msgr at 2023-09-30T00:53:05.434 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 1 librados: starting objecter 2023-09-30T00:53:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 1 Processor -- start 2023-09-30T00:53:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 1 -- start start 2023-09-30T00:53:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 1 librados: setting wanted keys 2023-09-30T00:53:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 1 librados: calling monclient init 2023-09-30T00:53:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 10 monclient: init 2023-09-30T00:53:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:53:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a680a5710 0x7f6a680aaae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 10 monclient(hunting): picked mon.noname-a con 0x7f6a680a5710 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6813e1d0 0x7f6a680ad940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 10 monclient(hunting): picked mon.noname-c con 0x7f6a6813e1d0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a6813eb60 0x7f6a680ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 10 monclient(hunting): picked mon.noname-b con 0x7f6a6813eb60 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68140190 con 0x7f6a680a5710 2023-09-30T00:53:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68140470 con 0x7f6a6813eb60 2023-09-30T00:53:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68140750 con 0x7f6a6813e1d0 2023-09-30T00:53:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.427+0000 7f6a6d325700 10 monclient(hunting): authenticate will time out at 5646.729000s 2023-09-30T00:53:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.428+0000 7f6a667fc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6813e1d0 0x7f6a680ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.428+0000 7f6a677fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a6813eb60 0x7f6a680ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:05.436 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.428+0000 7f6a667fc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6813e1d0 0x7f6a680ad940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:54886/0 (socket says 172.21.15.40:54886) 2023-09-30T00:53:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.428+0000 7f6a667fc700 1 -- 172.21.15.40:0/3273090596 learned_addr learned my addr 172.21.15.40:0/3273090596 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.428+0000 7f6a667fc700 10 monclient(hunting): get_auth_request con 0x7f6a6813e1d0 auth_method 0 2023-09-30T00:53:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.428+0000 7f6a667fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.428+0000 7f6a667fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.428+0000 7f6a667fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.428+0000 7f6a66ffd700 1 --2- 172.21.15.40:0/3273090596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a680a5710 0x7f6a680aaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.429+0000 7f6a677fe700 10 monclient(hunting): get_auth_request con 0x7f6a6813eb60 auth_method 0 2023-09-30T00:53:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.429+0000 7f6a677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.429+0000 7f6a677fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.429+0000 7f6a677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.429+0000 7f6a66ffd700 10 monclient(hunting): get_auth_request con 0x7f6a680a5710 auth_method 0 2023-09-30T00:53:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.429+0000 7f6a66ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:05.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.429+0000 7f6a66ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:05.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.429+0000 7f6a66ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:05.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.429+0000 7f6a667fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:05.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.429+0000 7f6a667fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:05.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.429+0000 7f6a667fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:05.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.429+0000 7f6a677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:05.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.429+0000 7f6a677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:05.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.429+0000 7f6a677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:05.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.430+0000 7f6a66ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:05.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.430+0000 7f6a66ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:05.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.430+0000 7f6a66ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:05.438 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.433+0000 7f6a677fe700 10 monclient(hunting): handle_auth_done global_id 4870 payload 824 2023-09-30T00:53:05.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.433+0000 7f6a677fe700 1 -- 172.21.15.40:0/3273090596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6813e1d0 msgr2=0x7f6a680ad940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:05.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.433+0000 7f6a677fe700 1 --2- 172.21.15.40:0/3273090596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6813e1d0 0x7f6a680ad940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.433+0000 7f6a677fe700 1 -- 172.21.15.40:0/3273090596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a680a5710 msgr2=0x7f6a680aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:05.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.433+0000 7f6a677fe700 1 --2- 172.21.15.40:0/3273090596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a680a5710 0x7f6a680aaae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.433+0000 7f6a677fe700 10 monclient: _finish_hunting 0 2023-09-30T00:53:05.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.433+0000 7f6a677fe700 1 monclient: found mon.noname-b 2023-09-30T00:53:05.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.433+0000 7f6a677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:05.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.433+0000 7f6a677fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:05.439 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.433+0000 7f6a677fe700 1 -- 172.21.15.40:0/3273090596 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a60009ae0 con 0x7f6a6813eb60 2023-09-30T00:53:05.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.435+0000 7f6a677fe700 1 --2- 172.21.15.40:0/3273090596 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a6813eb60 0x7f6a680ade80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6a60014000 tx=0x7f6a60017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:05.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.435+0000 7f6a4effd700 10 client.?.objecter ms_handle_connect 0x7f6a6813eb60 2023-09-30T00:53:05.440 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.435+0000 7f6a4effd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:05.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a4effd700 1 -- 172.21.15.40:0/3273090596 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a600032f0 con 0x7f6a6813eb60 2023-09-30T00:53:05.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a4effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:05.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a4effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:05.441 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a4effd700 10 monclient: dump: 2023-09-30T00:53:05.441 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:05.441 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:05.441 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:05.441 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:05.442 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a4effd700 10 monclient: _finish_auth 0 2023-09-30T00:53:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a4effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a4effd700 1 -- 172.21.15.40:0/3273090596 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a60012d00 con 0x7f6a6813eb60 2023-09-30T00:53:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a6d325700 5 monclient: authenticate success, global_id 4870 2023-09-30T00:53:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a6d325700 10 monclient: _renew_subs 2023-09-30T00:53:05.442 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a6d325700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:05.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a6d325700 1 -- 172.21.15.40:0/3273090596 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a68151b40 con 0x7f6a6813eb60 2023-09-30T00:53:05.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a4effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:05.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a4effd700 1 -- 172.21.15.40:0/3273090596 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6001e6a0 con 0x7f6a6813eb60 2023-09-30T00:53:05.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a4effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:05.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a4effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:05.443 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a4effd700 10 monclient: dump: 2023-09-30T00:53:05.443 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:05.443 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:05.443 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:05.443 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:05.443 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:05.443 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:05.443 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:05.444 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:05.444 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:05.444 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:05.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a657fa700 4 set_mon_vals no callback set 2023-09-30T00:53:05.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a6d325700 10 client.4870.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:05.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a6d325700 10 monclient: _renew_subs 2023-09-30T00:53:05.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a6d325700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:05.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a6d325700 1 -- 172.21.15.40:0/3273090596 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a681515b0 con 0x7f6a6813eb60 2023-09-30T00:53:05.444 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.440+0000 7f6a6d325700 1 librados: init done 2023-09-30T00:53:05.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.441+0000 7f6a6d325700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:05.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.441+0000 7f6a6d325700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:05.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.441+0000 7f6a6d325700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:05.445 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.441+0000 7f6a6d325700 1 -- 172.21.15.40:0/3273090596 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a68151eb0 con 0x7f6a6813eb60 2023-09-30T00:53:05.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.461+0000 7f6a4effd700 1 -- 172.21.15.40:0/3273090596 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6a6001ea50 con 0x7f6a6813eb60 2023-09-30T00:53:05.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.461+0000 7f6a4effd700 1 --2- 172.21.15.40:0/3273090596 >> v2:172.21.15.135:6800/116426 conn(0x7f6a54068960 0x7f6a5406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:05.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.461+0000 7f6a66ffd700 1 --2- 172.21.15.40:0/3273090596 >> v2:172.21.15.135:6800/116426 conn(0x7f6a54068960 0x7f6a5406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:05.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.461+0000 7f6a66ffd700 10 monclient: get_auth_request con 0x7f6a54068960 auth_method 0 2023-09-30T00:53:05.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.462+0000 7f6a66ffd700 1 --2- 172.21.15.40:0/3273090596 >> v2:172.21.15.135:6800/116426 conn(0x7f6a54068960 0x7f6a5406ae10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6a58003a30 tx=0x7f6a58007ea0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:05.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.462+0000 7f6a4effd700 10 client.4870.objecter ms_handle_connect 0x7f6a54068960 2023-09-30T00:53:05.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.691+0000 7f6a4effd700 1 -- 172.21.15.40:0/3273090596 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6a60027ea0 con 0x7f6a6813eb60 2023-09-30T00:53:05.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.691+0000 7f6a4effd700 10 client.4870.objecter ms_dispatch 0x7f6a68001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:05.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.691+0000 7f6a4effd700 3 client.4870.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:05.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.691+0000 7f6a4effd700 3 client.4870.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:05.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.691+0000 7f6a4effd700 20 client.4870.objecter dump_active .. 0 homeless 2023-09-30T00:53:05.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.702+0000 7f6a4effd700 1 -- 172.21.15.40:0/3273090596 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6a60053cf0 con 0x7f6a6813eb60 2023-09-30T00:53:05.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.702+0000 7f6a4effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:05.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.702+0000 7f6a4effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:05.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.911+0000 7f6a6d325700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:53:05.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.911+0000 7f6a6d325700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:53:05.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.911+0000 7f6a6d325700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:05.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.911+0000 7f6a6d325700 1 -- 172.21.15.40:0/3273090596 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a68072b90 con 0x7f6a6813eb60 2023-09-30T00:53:05.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.940+0000 7f6a4effd700 1 -- 172.21.15.40:0/3273090596 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+11935 (secure 0 0 0) 0x7f6a60057310 con 0x7f6a6813eb60 2023-09-30T00:53:05.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.940+0000 7f6a4effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:53:05.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.940+0000 7f6a4effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:53:05.940 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:53:05.941 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"fsid":"238dba7c-9190-48bd-bb67-322ebc9ecbd0","created":"2023-09-30T00:46:45.822318+0000","modified":"2023-09-30T00:47:55.738588+0000","last_up_change":"2023-09-30T00:47:45.793188+0000","last_in_change":"2023-09-30T00:46:52.657791+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:47:40.523068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:47:51.796955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"66e1da65-310e-4463-8aa0-431896832e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":117238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":117238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":117238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":117238}]},"public_addr":"172.21.15.40:6804/117238","cluster_addr":"172.21.15.40:6805/117238","heartbeat_back_addr":"172.21.15.40:6807/117238","heartbeat_front_addr":"172.21.15.40:6806/117238","state":["exists","up"]},{"osd":1,"uuid":"d6dd269f-2eda-4ef7-9da1-2ed5b16291b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":117237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6801","nonce":117237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":117237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":117237}]},"public_addr":"172.21.15.40:6800/117237","cluster_addr":"172.21.15.40:6801/117237","heartbeat_back_addr":"172.21.15.40:6803/117237","heartbeat_front_addr":"172.21.15.40:6802/117237","state":["exists","up"]},{"osd":2,"uuid":"8f0de27d-3674-4ff4-8451-4fcc90786c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":117239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":117239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":117239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":117239}]},"public_addr":"172.21.15.40:6808/117239","cluster_addr":"172.21.15.40:6809/117239","heartbeat_back_addr":"172.21.15.40:6811/117239","heartbeat_front_addr":"172.21.15.40:6810/117239","state":["exists","up"]},{"osd":3,"uuid":"d18ec2f5-7e8f-4732-a523-adc495115cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":117236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":117236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":117236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":117236}]},"public_addr":"172.21.15.40:6812/117236","cluster_addr":"172.21.15.40:6813/117236","heartbeat_back_addr":"172.21.15.40:6815/117236","heartbeat_front_addr":"172.21.15.40:6814/117236","state":["exists","up"]},{"osd":4,"uuid":"4e2465fe-bb27-47bd-8ddf-f334cac6041a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6805","nonce":117131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":117131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":117131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6807","nonce":117131}]},"public_addr":"172.21.15.135:6805/117131","cluster_addr":"172.21.15.135:6806/117131","heartbeat_back_addr":"172.21.15.135:6808/117131","heartbeat_front_addr":"172.21.15.135:6807/117131","state":["exists","up"]},{"osd":5,"uuid":"7ea449db-9b22-48b8-a61b-1937eeeb6c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6801","nonce":117130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":117130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":117130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6803","nonce":117130}]},"public_addr":"172.21.15.135:6801/117130","cluster_addr":"172.21.15.135:6802/117130","heartbeat_back_addr":"172.21.15.135:6804/117130","heartbeat_front_addr":"172.21.15.135:6803/117130","state":["exists","up"]},{"osd":6,"uuid":"c79b4829-6393-4f9a-a9f8-d2aacb505ae9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6813","nonce":117133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":117133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":117133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6815","nonce":117133}]},"public_addr":"172.21.15.135:6813/117133","cluster_addr":"172.21.15.135:6814/117133","heartbeat_back_addr":"172.21.15.135:6816/117133","heartbeat_front_addr":"172.21.15.135:6815/117133","state":["exists","up"]},{"osd":7,"uuid":"4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6809","nonce":117132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":117132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":117132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6811","nonce":117132}]},"public_addr":"172.21.15.135:6809/117132","cluster_addr":"172.21.15.135:6810/117132","heartbeat_back_addr":"172.21.15.135:6812/117132","heartbeat_front_addr":"172.21.15.135:6811/117132","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.725062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.291589+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.713534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.731764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:35.977051+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:36.680016+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:35.693147+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:36.368009+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:53:05.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 10 librados: watch_flush enter 2023-09-30T00:53:05.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 10 librados: watch_flush exit 2023-09-30T00:53:05.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 20 client.4870.objecter shutdown clearing up homeless session... 2023-09-30T00:53:05.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 10 client.4870.objecter successfully canceled tick 2023-09-30T00:53:05.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 1 -- 172.21.15.40:0/3273090596 >> v2:172.21.15.135:6800/116426 conn(0x7f6a54068960 msgr2=0x7f6a5406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 1 --2- 172.21.15.40:0/3273090596 >> v2:172.21.15.135:6800/116426 conn(0x7f6a54068960 0x7f6a5406ae10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6a58003a30 tx=0x7f6a58007ea0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 10 monclient: shutdown 2023-09-30T00:53:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 1 -- 172.21.15.40:0/3273090596 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a6813eb60 msgr2=0x7f6a680ade80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 1 --2- 172.21.15.40:0/3273090596 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a6813eb60 0x7f6a680ade80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6a60014000 tx=0x7f6a60017000 comp rx=0 tx=0).stop 2023-09-30T00:53:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 1 -- 172.21.15.40:0/3273090596 shutdown_connections 2023-09-30T00:53:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 1 --2- 172.21.15.40:0/3273090596 >> v2:172.21.15.135:6800/116426 conn(0x7f6a54068960 0x7f6a5406ae10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 1 --2- 172.21.15.40:0/3273090596 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f6a6813eb60 0x7f6a680ade80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 1 --2- 172.21.15.40:0/3273090596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a680a5710 0x7f6a680aaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 1 --2- 172.21.15.40:0/3273090596 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f6a6813e1d0 0x7f6a680ad940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 1 -- 172.21.15.40:0/3273090596 >> 172.21.15.40:0/3273090596 conn(0x7f6a68069080 msgr2=0x7f6a680b0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 1 -- 172.21.15.40:0/3273090596 shutdown_connections 2023-09-30T00:53:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 1 -- 172.21.15.40:0/3273090596 wait complete. 2023-09-30T00:53:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:05.942+0000 7f6a6d325700 1 librados: shutdown 2023-09-30T00:53:05.957 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T00:53:05.957 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:53:06.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.087+0000 7fa810b7a700 10 monclient: get_monmap_and_config 2023-09-30T00:53:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.087+0000 7fa810b7a700 10 monclient: build_initial_monmap 2023-09-30T00:53:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.087+0000 7fa810b7a700 10 monclient: monmap: 2023-09-30T00:53:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:06.088439+0000 2023-09-30T00:53:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:06.088439+0000 2023-09-30T00:53:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:06.088 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:06.089 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:06.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 1 Processor -- start 2023-09-30T00:53:06.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 1 -- start start 2023-09-30T00:53:06.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 10 monclient: init 2023-09-30T00:53:06.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-09-30T00:53:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa80c0b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:53:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa80c0b1830 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fa80c0a5fb0 0x7fa80c13b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 10 monclient(hunting): picked mon.noname-c con 0x7fa80c0a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80c0a5b20 con 0x7fa80c0b1460 2023-09-30T00:53:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80c0a71f0 con 0x7fa80c0b1830 2023-09-30T00:53:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa80c141e40 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.088+0000 7fa810b7a700 10 monclient(hunting): authenticate will time out at 5647.389954s 2023-09-30T00:53:06.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.089+0000 7fa80a59c700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa80c0b1460 0x7fa80c141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:54358/0 (socket says 172.21.15.40:54358) 2023-09-30T00:53:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.089+0000 7fa80a59c700 1 -- 172.21.15.40:0/30337663 learned_addr learned my addr 172.21.15.40:0/30337663 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:53:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.089+0000 7fa80ad9d700 1 --2- 172.21.15.40:0/30337663 >> v2:172.21.15.135:3300/0 conn(0x7fa80c0a5fb0 0x7fa80c13b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.090+0000 7fa80ad9d700 10 monclient(hunting): get_auth_request con 0x7fa80c0a5fb0 auth_method 0 2023-09-30T00:53:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.090+0000 7fa80ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.090+0000 7fa80ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.090+0000 7fa80ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.091+0000 7fa80ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.091+0000 7fa80ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.091+0000 7fa80ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.093+0000 7fa80ad9d700 10 monclient(hunting): handle_auth_done global_id 4873 payload 274 2023-09-30T00:53:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.093+0000 7fa80ad9d700 1 -- 172.21.15.40:0/30337663 >> v1:172.21.15.135:6789/0 conn(0x7fa80c0b1830 legacy=0x7fa80c13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.093+0000 7fa80ad9d700 1 -- 172.21.15.40:0/30337663 >> v1:172.21.15.40:6789/0 conn(0x7fa80c0b1460 legacy=0x7fa80c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.093+0000 7fa80ad9d700 10 monclient: _finish_hunting 0 2023-09-30T00:53:06.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.093+0000 7fa80ad9d700 1 monclient: found mon.noname-c 2023-09-30T00:53:06.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.093+0000 7fa80ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:06.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.093+0000 7fa80ad9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:53:06.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.093+0000 7fa80ad9d700 1 -- 172.21.15.40:0/30337663 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80c0a74d0 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.095+0000 7fa80ad9d700 1 --2- 172.21.15.40:0/30337663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa80c0a5fb0 0x7fa80c13b330 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa804009e90 tx=0x7fa804009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afde7d41306b3960 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:06.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa80959a700 1 -- 172.21.15.40:0/30337663 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80400e070 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa80959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:06.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa80959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:53:06.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa80959a700 10 monclient: dump: 2023-09-30T00:53:06.101 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:06.101 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:06.101 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.101 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.101 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:06.101 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:06.101 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:06.101 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:06.101 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:06.101 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:06.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa80959a700 10 monclient: _finish_auth 0 2023-09-30T00:53:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa80959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa80959a700 1 -- 172.21.15.40:0/30337663 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80400bbb0 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa80959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa80959a700 1 -- 172.21.15.40:0/30337663 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa804012430 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa80959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa80959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa80959a700 10 monclient: dump: 2023-09-30T00:53:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.102 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:06.103 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:06.103 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:06.103 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:06.103 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:06.103 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:06.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa810b7a700 5 monclient: authenticate success, global_id 4873 2023-09-30T00:53:06.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa810b7a700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:06.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa810b7a700 4 set_mon_vals no callback set 2023-09-30T00:53:06.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.100+0000 7fa810b7a700 10 monclient: shutdown 2023-09-30T00:53:06.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.101+0000 7fa810b7a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:06.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.101+0000 7fa810b7a700 1 -- 172.21.15.40:0/30337663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa80c0a5fb0 msgr2=0x7fa80c13b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:06.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.101+0000 7fa810b7a700 1 --2- 172.21.15.40:0/30337663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa80c0a5fb0 0x7fa80c13b330 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa804009e90 tx=0x7fa804009780 comp rx=0 tx=0).stop 2023-09-30T00:53:06.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.101+0000 7fa810b7a700 1 -- 172.21.15.40:0/30337663 shutdown_connections 2023-09-30T00:53:06.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.101+0000 7fa810b7a700 1 --2- 172.21.15.40:0/30337663 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa80c0a5fb0 0x7fa80c13b330 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:06.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.101+0000 7fa810b7a700 1 -- 172.21.15.40:0/30337663 >> 172.21.15.40:0/30337663 conn(0x7fa80c0631f0 msgr2=0x7fa80c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:06.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.101+0000 7fa810b7a700 1 -- 172.21.15.40:0/30337663 shutdown_connections 2023-09-30T00:53:06.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.101+0000 7fa810b7a700 1 -- 172.21.15.40:0/30337663 wait complete. 2023-09-30T00:53:06.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.101+0000 7fa810b7a700 10 monclient: build_initial_monmap 2023-09-30T00:53:06.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.101+0000 7fa810b7a700 10 monclient: monmap: 2023-09-30T00:53:06.107 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:06.107 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:06.107 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:06.107 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:06.107 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:06.107 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:06.107 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:06.107 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:06.108 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:06.108 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:06.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 1 librados: starting msgr at 2023-09-30T00:53:06.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 1 librados: starting objecter 2023-09-30T00:53:06.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 1 Processor -- start 2023-09-30T00:53:06.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 1 -- start start 2023-09-30T00:53:06.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 1 librados: setting wanted keys 2023-09-30T00:53:06.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 1 librados: calling monclient init 2023-09-30T00:53:06.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 10 monclient: init 2023-09-30T00:53:06.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:06.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:53:06.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa80c0a5fb0 0x7fa80c0a9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:06.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 10 monclient(hunting): picked mon.noname-b con 0x7fa80c0a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa80c0b1460 0x7fa80c0aa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 10 monclient(hunting): picked mon.noname-c con 0x7fa80c0b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa80c0b1830 0x7fa80c0ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 10 monclient(hunting): picked mon.noname-a con 0x7fa80c0b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c0abab0 con 0x7fa80c0b1830 2023-09-30T00:53:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c0a9d30 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c0abce0 con 0x7fa80c0b1460 2023-09-30T00:53:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.102+0000 7fa810b7a700 10 monclient(hunting): authenticate will time out at 5647.403844s 2023-09-30T00:53:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.103+0000 7fa80a59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa80c0a5fb0 0x7fa80c0a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:06.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.103+0000 7fa809d9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa80c0b1460 0x7fa80c0aa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:06.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.103+0000 7fa80a59c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa80c0a5fb0 0x7fa80c0a9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:33592/0 (socket says 172.21.15.40:33592) 2023-09-30T00:53:06.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.103+0000 7fa80a59c700 1 -- 172.21.15.40:0/2249027043 learned_addr learned my addr 172.21.15.40:0/2249027043 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:06.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.103+0000 7fa80a59c700 10 monclient(hunting): get_auth_request con 0x7fa80c0a5fb0 auth_method 0 2023-09-30T00:53:06.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.103+0000 7fa80a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:06.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.103+0000 7fa80a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:06.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.103+0000 7fa80a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:06.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.103+0000 7fa809d9b700 10 monclient(hunting): get_auth_request con 0x7fa80c0b1460 auth_method 0 2023-09-30T00:53:06.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.103+0000 7fa809d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:06.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.103+0000 7fa809d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:06.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.103+0000 7fa809d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.103+0000 7fa80ad9d700 1 --2- 172.21.15.40:0/2249027043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa80c0b1830 0x7fa80c0ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.104+0000 7fa80a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.104+0000 7fa80a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.104+0000 7fa80a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.104+0000 7fa80ad9d700 10 monclient(hunting): get_auth_request con 0x7fa80c0b1830 auth_method 0 2023-09-30T00:53:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.104+0000 7fa80ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.104+0000 7fa80ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.104+0000 7fa80ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.104+0000 7fa809d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.104+0000 7fa809d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.104+0000 7fa809d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.104+0000 7fa80ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:06.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.104+0000 7fa80ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:06.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.104+0000 7fa80ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:06.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.107+0000 7fa80a59c700 10 monclient(hunting): handle_auth_done global_id 4876 payload 824 2023-09-30T00:53:06.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.107+0000 7fa80a59c700 1 -- 172.21.15.40:0/2249027043 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa80c0b1460 msgr2=0x7fa80c0aa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:06.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.107+0000 7fa80a59c700 1 --2- 172.21.15.40:0/2249027043 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa80c0b1460 0x7fa80c0aa1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:06.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.107+0000 7fa80a59c700 1 -- 172.21.15.40:0/2249027043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa80c0b1830 msgr2=0x7fa80c0ae300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:06.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.107+0000 7fa80a59c700 1 --2- 172.21.15.40:0/2249027043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa80c0b1830 0x7fa80c0ae300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:06.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.107+0000 7fa80a59c700 10 monclient: _finish_hunting 0 2023-09-30T00:53:06.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.107+0000 7fa80a59c700 1 monclient: found mon.noname-b 2023-09-30T00:53:06.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.107+0000 7fa80a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:06.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.107+0000 7fa80a59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:06.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.107+0000 7fa80a59c700 1 -- 172.21.15.40:0/2249027043 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa804009a20 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.109+0000 7fa80a59c700 1 --2- 172.21.15.40:0/2249027043 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa80c0a5fb0 0x7fa80c0a9680 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00c7c0 tx=0x7fa7fc007f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:06.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.109+0000 7fa7fa7fc700 10 client.?.objecter ms_handle_connect 0x7fa80c0a5fb0 2023-09-30T00:53:06.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.109+0000 7fa7fa7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa7fa7fc700 1 -- 172.21.15.40:0/2249027043 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc005b30 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa7fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa7fa7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa7fa7fc700 10 monclient: dump: 2023-09-30T00:53:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:06.116 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:06.116 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:06.116 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:06.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa7fa7fc700 10 monclient: _finish_auth 0 2023-09-30T00:53:06.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa7fa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:06.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa7fa7fc700 1 -- 172.21.15.40:0/2249027043 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7fc005c90 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa7fa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:06.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa7fa7fc700 1 -- 172.21.15.40:0/2249027043 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc007320 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa7fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:06.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa7fa7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:06.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa7fa7fc700 10 monclient: dump: 2023-09-30T00:53:06.116 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:06.116 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:06.117 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.117 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.117 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:06.117 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:06.117 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:06.117 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:06.117 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:06.117 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:06.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa810b7a700 5 monclient: authenticate success, global_id 4876 2023-09-30T00:53:06.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa810b7a700 10 monclient: _renew_subs 2023-09-30T00:53:06.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa810b7a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:06.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa810b7a700 1 -- 172.21.15.40:0/2249027043 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80c0a8b10 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa808d99700 4 set_mon_vals no callback set 2023-09-30T00:53:06.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa810b7a700 10 client.4876.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:06.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa810b7a700 10 monclient: _renew_subs 2023-09-30T00:53:06.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa810b7a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:06.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa810b7a700 1 -- 172.21.15.40:0/2249027043 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa80c0aa730 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.113+0000 7fa810b7a700 1 librados: init done 2023-09-30T00:53:06.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.114+0000 7fa810b7a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:06.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.114+0000 7fa810b7a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:06.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.114+0000 7fa810b7a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:06.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.114+0000 7fa810b7a700 1 -- 172.21.15.40:0/2249027043 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80c0b0fe0 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.133+0000 7fa7fa7fc700 1 -- 172.21.15.40:0/2249027043 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa7fc007690 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.133+0000 7fa7fa7fc700 1 --2- 172.21.15.40:0/2249027043 >> v2:172.21.15.135:6800/116426 conn(0x7fa7ec0688c0 0x7fa7ec06ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:06.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.134+0000 7fa809d9b700 1 --2- 172.21.15.40:0/2249027043 >> v2:172.21.15.135:6800/116426 conn(0x7fa7ec0688c0 0x7fa7ec06ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:06.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.134+0000 7fa809d9b700 10 monclient: get_auth_request con 0x7fa7ec0688c0 auth_method 0 2023-09-30T00:53:06.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.134+0000 7fa7fa7fc700 10 client.4876.objecter ms_handle_connect 0x7fa7ec0688c0 2023-09-30T00:53:06.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.134+0000 7fa809d9b700 1 --2- 172.21.15.40:0/2249027043 >> v2:172.21.15.135:6800/116426 conn(0x7fa7ec0688c0 0x7fa7ec06ad70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa80c13a410 tx=0x7fa800009060 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:06.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.367+0000 7fa7fa7fc700 1 -- 172.21.15.40:0/2249027043 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa7fc0553f0 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.367 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.367+0000 7fa7fa7fc700 10 client.4876.objecter ms_dispatch 0x7fa80c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:06.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.367+0000 7fa7fa7fc700 3 client.4876.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:06.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.367+0000 7fa7fa7fc700 3 client.4876.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:06.368 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.367+0000 7fa7fa7fc700 20 client.4876.objecter dump_active .. 0 homeless 2023-09-30T00:53:06.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.378+0000 7fa7fa7fc700 1 -- 172.21.15.40:0/2249027043 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa7fc051030 con 0x7fa80c0a5fb0 2023-09-30T00:53:06.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.378+0000 7fa7fa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:06.378 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.378+0000 7fa7fa7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:06.604 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.604+0000 7fa810b7a700 1 -- 172.21.15.40:0/2249027043 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa80c064a00 con 0x7fa7ec0688c0 2023-09-30T00:53:06.607 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.607+0000 7fa7fa7fc700 1 -- 172.21.15.40:0/2249027043 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70274 (secure 0 0 0) 0x7fa80c064a00 con 0x7fa7ec0688c0 2023-09-30T00:53:06.608 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:53:06.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 10 librados: watch_flush enter 2023-09-30T00:53:06.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 10 librados: watch_flush exit 2023-09-30T00:53:06.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 20 client.4876.objecter shutdown clearing up homeless session... 2023-09-30T00:53:06.609 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 10 client.4876.objecter successfully canceled tick 2023-09-30T00:53:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 1 -- 172.21.15.40:0/2249027043 >> v2:172.21.15.135:6800/116426 conn(0x7fa7ec0688c0 msgr2=0x7fa7ec06ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 1 --2- 172.21.15.40:0/2249027043 >> v2:172.21.15.135:6800/116426 conn(0x7fa7ec0688c0 0x7fa7ec06ad70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa80c13a410 tx=0x7fa800009060 comp rx=0 tx=0).stop 2023-09-30T00:53:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 10 monclient: shutdown 2023-09-30T00:53:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 1 -- 172.21.15.40:0/2249027043 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa80c0a5fb0 msgr2=0x7fa80c0a9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 1 --2- 172.21.15.40:0/2249027043 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa80c0a5fb0 0x7fa80c0a9680 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00c7c0 tx=0x7fa7fc007f30 comp rx=0 tx=0).stop 2023-09-30T00:53:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 1 -- 172.21.15.40:0/2249027043 shutdown_connections 2023-09-30T00:53:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 1 --2- 172.21.15.40:0/2249027043 >> v2:172.21.15.135:6800/116426 conn(0x7fa7ec0688c0 0x7fa7ec06ad70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 1 --2- 172.21.15.40:0/2249027043 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fa80c0a5fb0 0x7fa80c0a9680 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 1 --2- 172.21.15.40:0/2249027043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa80c0b1830 0x7fa80c0ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 1 --2- 172.21.15.40:0/2249027043 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fa80c0b1460 0x7fa80c0aa1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 1 -- 172.21.15.40:0/2249027043 >> 172.21.15.40:0/2249027043 conn(0x7fa80c0631f0 msgr2=0x7fa80c0635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 1 -- 172.21.15.40:0/2249027043 shutdown_connections 2023-09-30T00:53:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 1 -- 172.21.15.40:0/2249027043 wait complete. 2023-09-30T00:53:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.609+0000 7fa810b7a700 1 librados: shutdown 2023-09-30T00:53:06.612 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-09-30T00:53:06.623 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2023-09-30T00:53:05.938599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43464,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931256,"statfs":{"total":773094113280,"available":773049606144,"internally_reserved":0,"allocated":2506752,"data_stored":1383352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002786"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.785504+0000","last_change":"2023-09-30T00:47:54.691712+0000","last_active":"2023-09-30T00:47:54.785504+0000","last_peered":"2023-09-30T00:47:54.785504+0000","last_clean":"2023-09-30T00:47:54.785504+0000","last_became_active":"2023-09-30T00:47:54.672127+0000","last_became_peered":"2023-09-30T00:47:54.672127+0000","last_unstale":"2023-09-30T00:47:54.785504+0000","last_undegraded":"2023-09-30T00:47:54.785504+0000","last_fullsized":"2023-09-30T00:47:54.785504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:32.704584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.160955+0000","last_change":"2023-09-30T00:47:54.749019+0000","last_active":"2023-09-30T00:47:55.160955+0000","last_peered":"2023-09-30T00:47:55.160955+0000","last_clean":"2023-09-30T00:47:55.160955+0000","last_became_active":"2023-09-30T00:47:54.702773+0000","last_became_peered":"2023-09-30T00:47:54.702773+0000","last_unstale":"2023-09-30T00:47:55.160955+0000","last_undegraded":"2023-09-30T00:47:55.160955+0000","last_fullsized":"2023-09-30T00:47:55.160955+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:07:10.081341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.809132+0000","last_change":"2023-09-30T00:47:54.157108+0000","last_active":"2023-09-30T00:47:54.809132+0000","last_peered":"2023-09-30T00:47:54.809132+0000","last_clean":"2023-09-30T00:47:54.809132+0000","last_became_active":"2023-09-30T00:47:54.139464+0000","last_became_peered":"2023-09-30T00:47:54.139464+0000","last_unstale":"2023-09-30T00:47:54.809132+0000","last_undegraded":"2023-09-30T00:47:54.809132+0000","last_fullsized":"2023-09-30T00:47:54.809132+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:42:38.117935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.795839+0000","last_change":"2023-09-30T00:47:53.993668+0000","last_active":"2023-09-30T00:47:54.795839+0000","last_peered":"2023-09-30T00:47:54.795839+0000","last_clean":"2023-09-30T00:47:54.795839+0000","last_became_active":"2023-09-30T00:47:53.951882+0000","last_became_peered":"2023-09-30T00:47:53.951882+0000","last_unstale":"2023-09-30T00:47:54.795839+0000","last_undegraded":"2023-09-30T00:47:54.795839+0000","last_fullsized":"2023-09-30T00:47:54.795839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:15:02.278001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'77","reported_seq":114,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:47:57.719324+0000","last_change":"2023-09-30T00:47:47.347565+0000","last_active":"2023-09-30T00:47:57.719324+0000","last_peered":"2023-09-30T00:47:57.719324+0000","last_clean":"2023-09-30T00:47:57.719324+0000","last_became_active":"2023-09-30T00:47:46.406306+0000","last_became_peered":"2023-09-30T00:47:46.406306+0000","last_unstale":"2023-09-30T00:47:57.719324+0000","last_undegraded":"2023-09-30T00:47:57.719324+0000","last_fullsized":"2023-09-30T00:47:57.719324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:40.867840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:40.867840+0000","last_clean_scrub_stamp":"2023-09-30T00:47:40.867840+0000","objects_scrubbed":0,"log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:17:21.219288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.780151+0000","last_change":"2023-09-30T00:47:54.729272+0000","last_active":"2023-09-30T00:47:54.780151+0000","last_peered":"2023-09-30T00:47:54.780151+0000","last_clean":"2023-09-30T00:47:54.780151+0000","last_became_active":"2023-09-30T00:47:54.710752+0000","last_became_peered":"2023-09-30T00:47:54.710752+0000","last_unstale":"2023-09-30T00:47:54.780151+0000","last_undegraded":"2023-09-30T00:47:54.780151+0000","last_fullsized":"2023-09-30T00:47:54.780151+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:59:21.956005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.239361+0000","last_change":"2023-09-30T00:47:53.773125+0000","last_active":"2023-09-30T00:47:55.239361+0000","last_peered":"2023-09-30T00:47:55.239361+0000","last_clean":"2023-09-30T00:47:55.239361+0000","last_became_active":"2023-09-30T00:47:53.761460+0000","last_became_peered":"2023-09-30T00:47:53.761460+0000","last_unstale":"2023-09-30T00:47:55.239361+0000","last_undegraded":"2023-09-30T00:47:55.239361+0000","last_fullsized":"2023-09-30T00:47:55.239361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:55.538707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.791094+0000","last_change":"2023-09-30T00:47:54.294197+0000","last_active":"2023-09-30T00:47:54.791094+0000","last_peered":"2023-09-30T00:47:54.791094+0000","last_clean":"2023-09-30T00:47:54.791094+0000","last_became_active":"2023-09-30T00:47:54.279706+0000","last_became_peered":"2023-09-30T00:47:54.279706+0000","last_unstale":"2023-09-30T00:47:54.791094+0000","last_undegraded":"2023-09-30T00:47:54.791094+0000","last_fullsized":"2023-09-30T00:47:54.791094+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:24:53.339591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.240181+0000","last_change":"2023-09-30T00:47:54.352713+0000","last_active":"2023-09-30T00:47:55.240181+0000","last_peered":"2023-09-30T00:47:55.240181+0000","last_clean":"2023-09-30T00:47:55.240181+0000","last_became_active":"2023-09-30T00:47:54.305507+0000","last_became_peered":"2023-09-30T00:47:54.305507+0000","last_unstale":"2023-09-30T00:47:55.240181+0000","last_undegraded":"2023-09-30T00:47:55.240181+0000","last_fullsized":"2023-09-30T00:47:55.240181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:21:08.873800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345988,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":13.013999999999999,"5min":12.769,"15min":13.343},"min":{"1min":7.7699999999999996,"5min":6.6319999999999997,"15min":6.6319999999999997},"max":{"1min":20.431999999999999,"5min":26.166,"15min":26.166},"last":7.4299999999999997},{"interface":"front","average":{"1min":10.711,"5min":11.778,"15min":13.131},"min":{"1min":6.8440000000000003,"5min":6.8440000000000003,"15min":6.8440000000000003},"max":{"1min":15.452,"5min":32.808,"15min":32.808},"last":9.734}]},{"osd":1,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":12.897,"5min":13.236000000000001,"15min":14.891999999999999},"min":{"1min":9.0199999999999996,"5min":7.8250000000000002,"15min":7.8250000000000002},"max":{"1min":21.581,"5min":31.925999999999998,"15min":31.925999999999998},"last":11.193},{"interface":"front","average":{"1min":15.115,"5min":16.175999999999998,"15min":17.780000000000001},"min":{"1min":9.3369999999999997,"5min":8.4629999999999992,"15min":8.4629999999999992},"max":{"1min":23.300000000000001,"5min":42.737000000000002,"15min":42.737000000000002},"last":12.452999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":14.624000000000001,"5min":16.120999999999999,"15min":17.459},"min":{"1min":8.9489999999999998,"5min":8.3469999999999995,"15min":8.3469999999999995},"max":{"1min":22.853000000000002,"5min":36.436999999999998,"15min":36.436999999999998},"last":15.381},{"interface":"front","average":{"1min":18.524999999999999,"5min":19.745999999999999,"15min":25.524999999999999},"min":{"1min":11.426,"5min":10.092000000000001,"15min":10.092000000000001},"max":{"1min":22.728999999999999,"5min":164.05600000000001,"15min":164.05600000000001},"last":16.114000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":15.416,"5min":17.084,"15min":17.538},"min":{"1min":10.166,"5min":8.9570000000000007,"15min":8.9570000000000007},"max":{"1min":21.091999999999999,"5min":36.740000000000002,"15min":36.740000000000002},"last":13.535},{"interface":"front","average":{"1min":13.753,"5min":15.188000000000001,"15min":17.27},"min":{"1min":8.1069999999999993,"5min":7.4329999999999998,"15min":7.4329999999999998},"max":{"1min":18.597999999999999,"5min":39.865000000000002,"15min":39.865000000000002},"last":21.573}]},{"osd":5,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":18.372,"5min":19.262,"15min":21.207000000000001},"min":{"1min":12.487,"5min":10.294,"15min":10.294},"max":{"1min":22.335999999999999,"5min":47.259,"15min":47.259},"last":18.552},{"interface":"front","average":{"1min":18.518000000000001,"5min":19.602,"15min":21.158000000000001},"min":{"1min":10.493,"5min":9.7059999999999995,"15min":9.7059999999999995},"max":{"1min":23.382999999999999,"5min":46.417999999999999,"15min":46.417999999999999},"last":20.602}]},{"osd":6,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":20.271000000000001,"5min":20.856999999999999,"15min":22.073},"min":{"1min":13.49,"5min":11.042999999999999,"15min":11.042999999999999},"max":{"1min":25.059999999999999,"5min":49.011000000000003,"15min":49.011000000000003},"last":14.67},{"interface":"front","average":{"1min":16.920999999999999,"5min":17.010999999999999,"15min":18.483000000000001},"min":{"1min":8.9130000000000003,"5min":8.9130000000000003,"15min":8.9130000000000003},"max":{"1min":24.98,"5min":44.369,"15min":44.369},"last":22.841999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":19.391999999999999,"15min":21.218},"min":{"1min":13.249000000000001,"5min":12.590999999999999,"15min":12.590999999999999},"max":{"1min":25.725000000000001,"5min":39.171999999999997,"15min":39.171999999999997},"last":23.925999999999998},{"interface":"front","average":{"1min":19.501999999999999,"5min":21.446999999999999,"15min":23.422000000000001},"min":{"1min":13.321999999999999,"5min":11.161,"15min":11.161},"max":{"1min":24.439,"5min":41.865000000000002,"15min":41.865000000000002},"last":16.963999999999999}]}]},{"osd":2,"up_from":20,"seq":85899345987,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":12.666,"5min":12.167,"15min":12.167},"min":{"1min":6.9450000000000003,"5min":6.6920000000000002,"15min":6.6920000000000002},"max":{"1min":20.530999999999999,"5min":22.748999999999999,"15min":22.748999999999999},"last":8.0329999999999995},{"interface":"front","average":{"1min":12.608000000000001,"5min":13.143000000000001,"15min":14.31},"min":{"1min":9.4100000000000001,"5min":6.319,"15min":6.319},"max":{"1min":21.957000000000001,"5min":31.885000000000002,"15min":31.885000000000002},"last":9.2189999999999994}]},{"osd":1,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":15.48,"5min":17.213000000000001,"15min":22.105},"min":{"1min":9.6579999999999995,"5min":8.718,"15min":8.718},"max":{"1min":21.058,"5min":125.67700000000001,"15min":125.67700000000001},"last":14.252000000000001},{"interface":"front","average":{"1min":16.795999999999999,"5min":17.684999999999999,"15min":24.204999999999998},"min":{"1min":10.210000000000001,"5min":8.0370000000000008,"15min":8.0370000000000008},"max":{"1min":24.420999999999999,"5min":156.821,"15min":156.821},"last":10.555999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":10.301,"5min":18.065999999999999,"15min":34.295999999999999},"min":{"1min":7.6210000000000004,"5min":7.6210000000000004,"15min":7.6210000000000004},"max":{"1min":14.249000000000001,"5min":513.32600000000002,"15min":513.32600000000002},"last":11.363},{"interface":"front","average":{"1min":17.056999999999999,"5min":24.167999999999999,"15min":42.537999999999997},"min":{"1min":11.009,"5min":10.308,"15min":10.308},"max":{"1min":23.140999999999998,"5min":598.48900000000003,"15min":598.48900000000003},"last":19.898}]},{"osd":4,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":14.541,"5min":16.052,"15min":17.638999999999999},"min":{"1min":9.1530000000000005,"5min":8.8859999999999992,"15min":8.8859999999999992},"max":{"1min":24.239999999999998,"5min":31.702000000000002,"15min":31.702000000000002},"last":12.669},{"interface":"front","average":{"1min":16.594000000000001,"5min":16.542000000000002,"15min":17.376999999999999},"min":{"1min":8.6750000000000007,"5min":8.6750000000000007,"15min":8.6750000000000007},"max":{"1min":23.922999999999998,"5min":37.073999999999998,"15min":37.073999999999998},"last":12.032}]},{"osd":5,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":19.826000000000001,"5min":20.279,"15min":21.254000000000001},"min":{"1min":13.452999999999999,"5min":9.1440000000000001,"15min":9.1440000000000001},"max":{"1min":25.805,"5min":38.981000000000002,"15min":38.981000000000002},"last":20.878},{"interface":"front","average":{"1min":18.370999999999999,"5min":20.248000000000001,"15min":23.968},"min":{"1min":13.335000000000001,"5min":9.8949999999999996,"15min":9.8949999999999996},"max":{"1min":24.638999999999999,"5min":74.510000000000005,"15min":74.510000000000005},"last":16.526}]},{"osd":6,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":18.266999999999999,"5min":17.795000000000002,"15min":19.167000000000002},"min":{"1min":13.561,"5min":7.6150000000000002,"15min":7.6150000000000002},"max":{"1min":24.135999999999999,"5min":40.875999999999998,"15min":40.875999999999998},"last":15.532999999999999},{"interface":"front","average":{"1min":22.277000000000001,"5min":21.561,"15min":23.045999999999999},"min":{"1min":15.621,"5min":9.5809999999999995,"15min":9.5809999999999995},"max":{"1min":26.98,"5min":42.539000000000001,"15min":42.539000000000001},"last":21.940999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":20.937000000000001,"5min":21.863,"15min":24.445},"min":{"1min":14.688000000000001,"5min":12.295999999999999,"15min":12.295999999999999},"max":{"1min":26.420999999999999,"5min":76.484999999999999,"15min":76.484999999999999},"last":17.858000000000001},{"interface":"front","average":{"1min":19.401,"5min":20.391999999999999,"15min":22.324000000000002},"min":{"1min":14.641999999999999,"5min":10.223000000000001,"15min":10.223000000000001},"max":{"1min":28.007999999999999,"5min":55.265999999999998,"15min":55.265999999999998},"last":18.835000000000001}]}]},{"osd":5,"up_from":15,"seq":64424509509,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":614400,"data_stored":471299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":10.923,"5min":10.705,"15min":11.093},"min":{"1min":7.1210000000000004,"5min":5.6130000000000004,"15min":5.6130000000000004},"max":{"1min":18.591999999999999,"5min":23.507000000000001,"15min":23.507000000000001},"last":14.978999999999999},{"interface":"front","average":{"1min":11.551,"5min":10.593999999999999,"15min":10.163},"min":{"1min":7.1470000000000002,"5min":5.2130000000000001,"15min":5.2130000000000001},"max":{"1min":20.713000000000001,"5min":27.425000000000001,"15min":27.425000000000001},"last":13.018000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.073,"5min":12.210000000000001,"15min":12.234},"min":{"1min":6.8369999999999997,"5min":6.6619999999999999,"15min":6.6619999999999999},"max":{"1min":15.596,"5min":31.512,"15min":31.512},"last":12.013},{"interface":"front","average":{"1min":16.539000000000001,"5min":19.786999999999999,"15min":26.213000000000001},"min":{"1min":11.753,"5min":10.006,"15min":10.006},"max":{"1min":21.158000000000001,"5min":232.38800000000001,"15min":232.38800000000001},"last":19.048999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":9.3290000000000006,"5min":10.186,"15min":9.9290000000000003},"min":{"1min":7.0999999999999996,"5min":6.9909999999999997,"15min":6.9909999999999997},"max":{"1min":15.125999999999999,"5min":22.271000000000001,"15min":22.271000000000001},"last":12.742000000000001},{"interface":"front","average":{"1min":16.318999999999999,"5min":16.466000000000001,"15min":17.452999999999999},"min":{"1min":12.247999999999999,"5min":8.3810000000000002,"15min":8.3810000000000002},"max":{"1min":19.457000000000001,"5min":30.27,"15min":30.27},"last":18.256}]},{"osd":3,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.276,"5min":13.022,"15min":14.42},"min":{"1min":7.1829999999999998,"5min":7.1829999999999998,"15min":7.1829999999999998},"max":{"1min":17.137,"5min":42.023000000000003,"15min":42.023000000000003},"last":8.1359999999999992},{"interface":"front","average":{"1min":14.127000000000001,"5min":14.695,"15min":16.593},"min":{"1min":9.5350000000000001,"5min":7.1029999999999998,"15min":7.1029999999999998},"max":{"1min":21.387,"5min":46.786000000000001,"15min":46.786000000000001},"last":10.852}]},{"osd":4,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":15.722,"5min":14.898,"15min":14.827999999999999},"min":{"1min":10.923,"5min":6.7380000000000004,"15min":6.7380000000000004},"max":{"1min":21.215,"5min":23.788,"15min":23.788},"last":16.582000000000001},{"interface":"front","average":{"1min":17.538,"5min":18.876000000000001,"15min":22.504000000000001},"min":{"1min":11.625,"5min":9.2300000000000004,"15min":9.2300000000000004},"max":{"1min":20.460000000000001,"5min":112.548,"15min":112.548},"last":20.884}]},{"osd":6,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":13.363,"5min":14.715999999999999,"15min":17.433},"min":{"1min":6.819,"5min":6.819,"15min":6.819},"max":{"1min":22.664000000000001,"5min":106.333,"15min":106.333},"last":17.274000000000001},{"interface":"front","average":{"1min":16.849,"5min":16.956,"15min":19.303000000000001},"min":{"1min":9.9529999999999994,"5min":8.2769999999999992,"15min":8.2769999999999992},"max":{"1min":23.702999999999999,"5min":103.483,"15min":103.483},"last":11.742000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":17.23,"5min":21.253,"15min":29.43},"min":{"1min":11.779999999999999,"5min":10.369999999999999,"15min":10.369999999999999},"max":{"1min":22.402999999999999,"5min":253.74799999999999,"15min":253.74799999999999},"last":15.756},{"interface":"front","average":{"1min":17.359999999999999,"5min":20.033999999999999,"15min":30.213999999999999},"min":{"1min":9.1869999999999994,"5min":9.0310000000000006,"15min":9.0310000000000006},"max":{"1min":21.940000000000001,"5min":328.31799999999998,"15min":328.31799999999998},"last":21.754000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509508,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":9.6129999999999995,"5min":10.558,"15min":12.276},"min":{"1min":6.4080000000000004,"5min":4.556,"15min":4.556},"max":{"1min":19.038,"5min":78.191000000000003,"15min":78.191000000000003},"last":13.297000000000001},{"interface":"front","average":{"1min":11.808,"5min":11.657,"15min":13.117000000000001},"min":{"1min":6.4189999999999996,"5min":5.5439999999999996,"15min":5.5439999999999996},"max":{"1min":21.591999999999999,"5min":73.787000000000006,"15min":73.787000000000006},"last":8.1829999999999998}]},{"osd":1,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":11.846,"5min":13.288,"15min":15.319000000000001},"min":{"1min":6.6219999999999999,"5min":6.6219999999999999,"15min":6.6219999999999999},"max":{"1min":19.167999999999999,"5min":79.558000000000007,"15min":79.558000000000007},"last":10.183},{"interface":"front","average":{"1min":13.247999999999999,"5min":14.374000000000001,"15min":16.331},"min":{"1min":7.1130000000000004,"5min":7.1130000000000004,"15min":7.1130000000000004},"max":{"1min":18.385999999999999,"5min":82.048000000000002,"15min":82.048000000000002},"last":8.5830000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":15.504,"5min":16.736999999999998,"15min":18.332000000000001},"min":{"1min":9.4640000000000004,"5min":8.2889999999999997,"15min":8.2889999999999997},"max":{"1min":20.145,"5min":98.372,"15min":98.372},"last":16.635999999999999},{"interface":"front","average":{"1min":14.375999999999999,"5min":15.202999999999999,"15min":16.692},"min":{"1min":8.0009999999999994,"5min":7.2750000000000004,"15min":7.2750000000000004},"max":{"1min":22.282,"5min":96.753,"15min":96.753},"last":14.579000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":16.495999999999999,"5min":17.460000000000001,"15min":19.975999999999999},"min":{"1min":9.0860000000000003,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":20.835000000000001,"5min":100.212,"15min":100.212},"last":14.289},{"interface":"front","average":{"1min":11.829000000000001,"5min":12.417,"15min":14.603},"min":{"1min":6.8029999999999999,"5min":6.7729999999999997,"15min":6.7729999999999997},"max":{"1min":19.068999999999999,"5min":69.975999999999999,"15min":69.975999999999999},"last":8.8279999999999994}]},{"osd":5,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":16.097000000000001,"5min":14.657999999999999,"15min":18.02},"min":{"1min":7.9880000000000004,"5min":7.3070000000000004,"15min":7.3070000000000004},"max":{"1min":20.745999999999999,"5min":129.88,"15min":129.88},"last":19.850999999999999},{"interface":"front","average":{"1min":15.327999999999999,"5min":18.414000000000001,"15min":21.858000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":22.792999999999999,"5min":101.876,"15min":101.876},"last":21.454999999999998}]},{"osd":6,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":19.172000000000001,"5min":21.908999999999999,"15min":28.599},"min":{"1min":14.031000000000001,"5min":11.853999999999999,"15min":11.853999999999999},"max":{"1min":23.716999999999999,"5min":199.161,"15min":199.161},"last":17.475999999999999},{"interface":"front","average":{"1min":13.936999999999999,"5min":17.481000000000002,"15min":24.242999999999999},"min":{"1min":9.1899999999999995,"5min":8.5060000000000002,"15min":8.5060000000000002},"max":{"1min":23.106000000000002,"5min":205.011,"15min":205.011},"last":15.991}]},{"osd":7,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":16.001000000000001,"5min":18.004999999999999,"15min":22.398},"min":{"1min":9.9570000000000007,"5min":8.2110000000000003,"15min":8.2110000000000003},"max":{"1min":22.222000000000001,"5min":140.71000000000001,"15min":140.71000000000001},"last":22.48},{"interface":"front","average":{"1min":16.390999999999998,"5min":17.071999999999999,"15min":19.221},"min":{"1min":10.544,"5min":5.6509999999999998,"15min":5.6509999999999998},"max":{"1min":23.492999999999999,"5min":101.80500000000001,"15min":101.80500000000001},"last":20.305}]}]},{"osd":6,"up_from":14,"seq":60129542213,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.535,"5min":13.951000000000001,"15min":22.998000000000001},"min":{"1min":6.5380000000000003,"5min":4.5990000000000002,"15min":4.5990000000000002},"max":{"1min":17.355,"5min":350.05700000000002,"15min":350.05700000000002},"last":7.2249999999999996},{"interface":"front","average":{"1min":9.7520000000000007,"5min":13.462999999999999,"15min":23.047999999999998},"min":{"1min":5.383,"5min":5.383,"15min":5.383},"max":{"1min":18.678000000000001,"5min":352.40899999999999,"15min":352.40899999999999},"last":9.1560000000000006}]},{"osd":1,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":12.308,"5min":18.155999999999999,"15min":33.904000000000003},"min":{"1min":6.367,"5min":5.5700000000000003,"15min":5.5700000000000003},"max":{"1min":19.513000000000002,"5min":332.67399999999998,"15min":332.67399999999998},"last":13.475},{"interface":"front","average":{"1min":12.640000000000001,"5min":19.193999999999999,"15min":37.317999999999998},"min":{"1min":7.8559999999999999,"5min":7.2629999999999999,"15min":7.2629999999999999},"max":{"1min":18.895,"5min":359.57900000000001,"15min":359.57900000000001},"last":16.504000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.337999999999999,"5min":18.093,"15min":29.870000000000001},"min":{"1min":6.907,"5min":6.593,"15min":6.593},"max":{"1min":22.527999999999999,"5min":420.69299999999998,"15min":420.69299999999998},"last":8.4580000000000002},{"interface":"front","average":{"1min":16.591999999999999,"5min":17.097000000000001,"15min":26.565000000000001},"min":{"1min":10.023,"5min":6.0990000000000002,"15min":6.0990000000000002},"max":{"1min":19.998000000000001,"5min":421.416,"15min":421.416},"last":17.954999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":13.147,"5min":14.747,"15min":17.933},"min":{"1min":7.4450000000000003,"5min":5.9710000000000001,"15min":5.9710000000000001},"max":{"1min":21.074999999999999,"5min":142.905,"15min":142.905},"last":11.849},{"interface":"front","average":{"1min":17.030999999999999,"5min":19.399999999999999,"15min":23.738},"min":{"1min":13.157,"5min":11.380000000000001,"15min":11.380000000000001},"max":{"1min":19.614000000000001,"5min":177.86199999999999,"15min":177.86199999999999},"last":19.448}]},{"osd":4,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":14.191000000000001,"5min":15.872999999999999,"15min":19.667999999999999},"min":{"1min":9.0670000000000002,"5min":7.0270000000000001,"15min":7.0270000000000001},"max":{"1min":20.523,"5min":164.02699999999999,"15min":164.02699999999999},"last":18.616},{"interface":"front","average":{"1min":17.466999999999999,"5min":19.172999999999998,"15min":23.913},"min":{"1min":10.384,"5min":6.7400000000000002,"15min":6.7400000000000002},"max":{"1min":22.350000000000001,"5min":183.18000000000001,"15min":183.18000000000001},"last":21.779}]},{"osd":5,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":15.390000000000001,"15min":19.122},"min":{"1min":7.5229999999999997,"5min":7.101,"15min":7.101},"max":{"1min":17.652000000000001,"5min":163.00700000000001,"15min":163.00700000000001},"last":15.446},{"interface":"front","average":{"1min":15.941000000000001,"5min":17.882000000000001,"15min":22.271999999999998},"min":{"1min":11.317,"5min":7.2770000000000001,"15min":7.2770000000000001},"max":{"1min":21.721,"5min":177.17400000000001,"15min":177.17400000000001},"last":20.670999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":18.608000000000001,"5min":20.460000000000001,"15min":24.350000000000001},"min":{"1min":12.318,"5min":8.8469999999999995,"15min":8.8469999999999995},"max":{"1min":21.919,"5min":179.86500000000001,"15min":179.86500000000001},"last":22.663},{"interface":"front","average":{"1min":14.218,"5min":17.603999999999999,"15min":21.802},"min":{"1min":7.4619999999999997,"5min":7.4619999999999997,"15min":7.4619999999999997},"max":{"1min":19.574000000000002,"5min":164.58099999999999,"15min":164.58099999999999},"last":16.140999999999998}]}]},{"osd":7,"up_from":15,"seq":64424509508,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":9.2799999999999994,"5min":10.220000000000001,"15min":11.925000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":17.898,"5min":52.545999999999999,"15min":52.545999999999999},"last":12.744},{"interface":"front","average":{"1min":10.803000000000001,"5min":11.276999999999999,"15min":13.035},"min":{"1min":7.5449999999999999,"5min":6.6180000000000003,"15min":6.6180000000000003},"max":{"1min":17.611000000000001,"5min":60.087000000000003,"15min":60.087000000000003},"last":10.69}]},{"osd":1,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":12.558999999999999,"5min":15.726000000000001,"15min":23.890999999999998},"min":{"1min":8.3209999999999997,"5min":5.5179999999999998,"15min":5.5179999999999998},"max":{"1min":18.693999999999999,"5min":292.69600000000003,"15min":292.69600000000003},"last":8.4469999999999992},{"interface":"front","average":{"1min":10.73,"5min":17.122,"15min":33.595999999999997},"min":{"1min":7.3499999999999996,"5min":6.9059999999999997,"15min":6.9059999999999997},"max":{"1min":19.689,"5min":571.03399999999999,"15min":571.03399999999999},"last":8.7119999999999997}]},{"osd":2,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":16.391999999999999,"5min":16.396999999999998,"15min":18.145},"min":{"1min":11.743,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":22.370000000000001,"5min":62.984000000000002,"15min":62.984000000000002},"last":15.945},{"interface":"front","average":{"1min":13.821999999999999,"5min":14.734,"15min":16.702000000000002},"min":{"1min":8.3650000000000002,"5min":7.875,"15min":7.875},"max":{"1min":21.393000000000001,"5min":59.280999999999999,"15min":59.280999999999999},"last":14.462999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":13.872999999999999,"5min":16.023,"15min":17.783000000000001},"min":{"1min":9.3239999999999998,"5min":7.8949999999999996,"15min":7.8949999999999996},"max":{"1min":21.843,"5min":65.084000000000003,"15min":65.084000000000003},"last":13.635},{"interface":"front","average":{"1min":16.149999999999999,"5min":14.801,"15min":16.591000000000001},"min":{"1min":8.1530000000000005,"5min":6.3650000000000002,"15min":6.3650000000000002},"max":{"1min":21.564,"5min":56.024999999999999,"15min":56.024999999999999},"last":17.960000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":14.815,"5min":15.505000000000001,"15min":16.719000000000001},"min":{"1min":9.6229999999999993,"5min":6.5369999999999999,"15min":6.5369999999999999},"max":{"1min":20.928000000000001,"5min":39.575000000000003,"15min":39.575000000000003},"last":15.119},{"interface":"front","average":{"1min":16.652999999999999,"5min":18.010000000000002,"15min":20.782},"min":{"1min":9.0199999999999996,"5min":8.5459999999999994,"15min":8.5459999999999994},"max":{"1min":20.765999999999998,"5min":113.193,"15min":113.193},"last":20.492000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":18.452000000000002,"5min":18.542000000000002,"15min":19.276},"min":{"1min":15.332000000000001,"5min":10.16,"15min":10.16},"max":{"1min":22.913,"5min":49.593000000000004,"15min":49.593000000000004},"last":21.762},{"interface":"front","average":{"1min":16.960999999999999,"5min":18.082000000000001,"15min":19.195},"min":{"1min":12.218999999999999,"5min":7.6820000000000004,"15min":7.6820000000000004},"max":{"1min":20.646000000000001,"5min":48.204000000000001,"15min":48.204000000000001},"last":19.838999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":18.614999999999998,"5min":19.957999999999998,"15min":23.268000000000001},"min":{"1min":14.460000000000001,"5min":10.625999999999999,"15min":10.625999999999999},"max":{"1min":23.527000000000001,"5min":115.434,"15min":115.434},"last":22.923999999999999},{"interface":"front","average":{"1min":17.231999999999999,"5min":16.93,"15min":17.43},"min":{"1min":12.894,"5min":8.0190000000000001,"15min":8.0190000000000001},"max":{"1min":22.411999999999999,"5min":40.805,"15min":40.805},"last":18.908999999999999}]}]},{"osd":0,"up_from":20,"seq":85899345988,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":12.696999999999999,"5min":13.567,"15min":14.483000000000001},"min":{"1min":5.0830000000000002,"5min":5.0830000000000002,"15min":5.0830000000000002},"max":{"1min":20.931000000000001,"5min":29.138999999999999,"15min":29.138999999999999},"last":8.5210000000000008},{"interface":"front","average":{"1min":11.182,"5min":12.558999999999999,"15min":13.718999999999999},"min":{"1min":6.9989999999999997,"5min":6.1059999999999999,"15min":6.1059999999999999},"max":{"1min":20.055,"5min":35.32,"15min":35.32},"last":12.678000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":9.7750000000000004,"5min":11.563000000000001,"15min":14.34},"min":{"1min":7.3109999999999999,"5min":6.9829999999999997,"15min":6.9829999999999997},"max":{"1min":14.356,"5min":56.628999999999998,"15min":56.628999999999998},"last":12.984},{"interface":"front","average":{"1min":16.048999999999999,"5min":17.128,"15min":19.702999999999999},"min":{"1min":11.364000000000001,"5min":10.457000000000001,"15min":10.457000000000001},"max":{"1min":20.681000000000001,"5min":50.646000000000001,"15min":50.646000000000001},"last":9.9819999999999993}]},{"osd":3,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":14.92,"5min":29.306999999999999,"15min":44.177},"min":{"1min":11.141,"5min":8.4399999999999995,"15min":8.4399999999999995},"max":{"1min":26.937000000000001,"5min":616.83500000000004,"15min":616.83500000000004},"last":16.058},{"interface":"front","average":{"1min":14.122999999999999,"5min":27.134,"15min":38.993000000000002},"min":{"1min":9.3360000000000003,"5min":9.1229999999999993,"15min":9.1229999999999993},"max":{"1min":22.553999999999998,"5min":495.32600000000002,"15min":495.32600000000002},"last":9.0950000000000006}]},{"osd":4,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":17.096,"5min":17.635999999999999,"15min":19.018999999999998},"min":{"1min":11.279999999999999,"5min":9.875,"15min":9.875},"max":{"1min":23.553999999999998,"5min":36.109000000000002,"15min":36.109000000000002},"last":17.013999999999999},{"interface":"front","average":{"1min":15.769,"5min":16.484999999999999,"15min":18.593},"min":{"1min":10.702,"5min":8.4420000000000002,"15min":8.4420000000000002},"max":{"1min":22.722999999999999,"5min":39.177999999999997,"15min":39.177999999999997},"last":15.233000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":18.515000000000001,"5min":19.923999999999999,"15min":22.416},"min":{"1min":12.333,"5min":9.3870000000000005,"15min":9.3870000000000005},"max":{"1min":26.995999999999999,"5min":66.313999999999993,"15min":66.313999999999993},"last":20.018000000000001},{"interface":"front","average":{"1min":20.390999999999998,"5min":22.050999999999998,"15min":23.927},"min":{"1min":12.085000000000001,"5min":12.085000000000001,"15min":12.085000000000001},"max":{"1min":25.503,"5min":41.634,"15min":41.634},"last":22.291}]},{"osd":6,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":14.702999999999999,"5min":17.117999999999999,"15min":22.646000000000001},"min":{"1min":10.412000000000001,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":21.728000000000002,"5min":54.033999999999999,"15min":54.033999999999999},"last":19.402000000000001},{"interface":"front","average":{"1min":20.824999999999999,"5min":22.52,"15min":24.59},"min":{"1min":16.77,"5min":11.055999999999999,"15min":11.055999999999999},"max":{"1min":24.806000000000001,"5min":69.688000000000002,"15min":69.688000000000002},"last":21.431999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":22.016999999999999,"5min":23.597999999999999,"15min":26.277000000000001},"min":{"1min":16.298999999999999,"5min":14.875,"15min":14.875},"max":{"1min":27.872,"5min":49.347999999999999,"15min":49.347999999999999},"last":23.643999999999998},{"interface":"front","average":{"1min":16.75,"5min":18.710999999999999,"15min":21.469000000000001},"min":{"1min":10.622,"5min":10.489000000000001,"15min":10.489000000000001},"max":{"1min":24.265000000000001,"5min":54.734999999999999,"15min":54.734999999999999},"last":14.289}]}]},{"osd":1,"up_from":19,"seq":81604378692,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6044,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365796,"statfs":{"total":96636764160,"available":96630575104,"internally_reserved":0,"allocated":614400,"data_stored":471299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":13.113,"15min":16.02},"min":{"1min":7.4809999999999999,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":13.616,"5min":91.582999999999998,"15min":91.582999999999998},"last":8.1799999999999997},{"interface":"front","average":{"1min":12.308,"5min":13.007999999999999,"15min":14.891999999999999},"min":{"1min":8.1780000000000008,"5min":7.2889999999999997,"15min":7.2889999999999997},"max":{"1min":26.073,"5min":75.853999999999999,"15min":75.853999999999999},"last":11.861000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":12.657999999999999,"5min":14.127000000000001,"15min":18.731000000000002},"min":{"1min":8.2639999999999993,"5min":8.2639999999999993,"15min":8.2639999999999993},"max":{"1min":20.702000000000002,"5min":142.41999999999999,"15min":142.41999999999999},"last":10.119999999999999},{"interface":"front","average":{"1min":15.305,"5min":16.303999999999998,"15min":19.814},"min":{"1min":9.952,"5min":8.0749999999999993,"15min":8.0749999999999993},"max":{"1min":23.823,"5min":133.988,"15min":133.988},"last":19.651}]},{"osd":3,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":14.644,"5min":17.609000000000002,"15min":23.707000000000001},"min":{"1min":8.9009999999999998,"5min":8.4019999999999992,"15min":8.4019999999999992},"max":{"1min":21.539000000000001,"5min":170.785,"15min":170.785},"last":12.593},{"interface":"front","average":{"1min":16.152000000000001,"5min":19.57,"15min":27.786000000000001},"min":{"1min":9.1349999999999998,"5min":9.1349999999999998,"15min":9.1349999999999998},"max":{"1min":23.108000000000001,"5min":170.17400000000001,"15min":170.17400000000001},"last":13.301}]},{"osd":4,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":15.271000000000001,"5min":18.404,"15min":24.379000000000001},"min":{"1min":10.272,"5min":8.9510000000000005,"15min":8.9510000000000005},"max":{"1min":19.027999999999999,"5min":108.045,"15min":108.045},"last":15.673},{"interface":"front","average":{"1min":17.634,"5min":20.315000000000001,"15min":27.440999999999999},"min":{"1min":10.829000000000001,"5min":9.9000000000000004,"15min":9.9000000000000004},"max":{"1min":23.558,"5min":141.97300000000001,"15min":141.97300000000001},"last":10.782999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":16.079000000000001,"5min":20.795999999999999,"15min":26.815999999999999},"min":{"1min":9.9489999999999998,"5min":9.9489999999999998,"15min":9.9489999999999998},"max":{"1min":22.390000000000001,"5min":116.873,"15min":116.873},"last":14.743},{"interface":"front","average":{"1min":19.495000000000001,"5min":22.710999999999999,"15min":31.870999999999999},"min":{"1min":11.393000000000001,"5min":9.9550000000000001,"15min":9.9550000000000001},"max":{"1min":24.968,"5min":199.773,"15min":199.773},"last":16.725999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":19.128,"5min":22.716000000000001,"15min":28.771999999999998},"min":{"1min":12.276999999999999,"5min":10.601000000000001,"15min":10.601000000000001},"max":{"1min":23.670999999999999,"5min":131.06299999999999,"15min":131.06299999999999},"last":17.422000000000001},{"interface":"front","average":{"1min":20.283000000000001,"5min":23.718,"15min":31.068000000000001},"min":{"1min":14.736000000000001,"5min":13.375999999999999,"15min":13.375999999999999},"max":{"1min":27.178000000000001,"5min":139.024,"15min":139.024},"last":18.157}]},{"osd":7,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":16.785,"5min":19.227,"15min":26.094999999999999},"min":{"1min":9.577,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":27.773,"5min":127.039,"15min":127.039},"last":20.626000000000001},{"interface":"front","average":{"1min":18.727,"5min":21.361999999999998,"15min":27.317},"min":{"1min":11.532,"5min":10.449,"15min":10.449},"max":{"1min":24.533999999999999,"5min":130.88499999999999,"15min":130.88499999999999},"last":21.276}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:53:06.625 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:53:06.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.763+0000 7efd43f1b700 10 monclient: get_monmap_and_config 2023-09-30T00:53:06.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.763+0000 7efd43f1b700 10 monclient: build_initial_monmap 2023-09-30T00:53:06.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.763+0000 7efd43f1b700 10 monclient: monmap: 2023-09-30T00:53:06.764 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:06.764 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:06.764 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:06.764428+0000 2023-09-30T00:53:06.765 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:06.764428+0000 2023-09-30T00:53:06.765 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:06.765 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:06.765 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:06.765 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:06.765 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:06.765 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:06.765 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:06.765 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:06.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.763+0000 7efd43f1b700 1 Processor -- start 2023-09-30T00:53:06.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 1 -- start start 2023-09-30T00:53:06.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 10 monclient: init 2023-09-30T00:53:06.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:06.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-09-30T00:53:06.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efd3c0b1460 0x7efd3c0b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:06.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 10 monclient(hunting): picked mon.noname-a con 0x7efd3c0b1460 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:06.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd3c0a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:06.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd3c13ac20 0x7efd3c13b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:06.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 10 monclient(hunting): picked mon.noname-b con 0x7efd3c13ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:06.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c0a5b20 con 0x7efd3c0a5fb0 2023-09-30T00:53:06.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:06.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd3c0a71f0 con 0x7efd3c0b1460 2023-09-30T00:53:06.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:06.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c0a74d0 con 0x7efd3c13ac20 2023-09-30T00:53:06.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:06.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.764+0000 7efd43f1b700 10 monclient(hunting): authenticate will time out at 5648.065938s 2023-09-30T00:53:06.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.765+0000 7efd424b8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd3c13ac20 0x7efd3c13b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:06.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.765+0000 7efd41cb7700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efd3c0b1460 0x7efd3c0b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:06.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.766+0000 7efd414b6700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7efd3c0a5fb0 0x7efd3c141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:40434/0 (socket says 172.21.15.40:40434) 2023-09-30T00:53:06.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.766+0000 7efd414b6700 1 -- 172.21.15.40:0/3656355395 learned_addr learned my addr 172.21.15.40:0/3656355395 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:53:06.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.766+0000 7efd424b8700 10 monclient(hunting): get_auth_request con 0x7efd3c13ac20 auth_method 0 2023-09-30T00:53:06.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.766+0000 7efd424b8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:06.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.766+0000 7efd424b8700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:06.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.766+0000 7efd424b8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:06.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.766+0000 7efd41cb7700 10 monclient(hunting): get_auth_request con 0x7efd3c0b1460 auth_method 0 2023-09-30T00:53:06.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.766+0000 7efd41cb7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:06.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.766+0000 7efd41cb7700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:06.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.766+0000 7efd41cb7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:06.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.767+0000 7efd424b8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:06.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.767+0000 7efd424b8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:06.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.767+0000 7efd424b8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:06.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.767+0000 7efd41cb7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:06.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.767+0000 7efd41cb7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:06.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.767+0000 7efd41cb7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:06.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.769+0000 7efd40cb5700 1 -- 172.21.15.40:0/3656355395 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537078140 0 0) 0x7efd3c0a5b20 con 0x7efd3c0a5fb0 2023-09-30T00:53:06.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.769+0000 7efd40cb5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:06.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.769+0000 7efd40cb5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:06.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.769+0000 7efd40cb5700 10 monclient(hunting): my global_id is 4879 2023-09-30T00:53:06.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.769+0000 7efd40cb5700 1 -- 172.21.15.40:0/3656355395 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd28003570 con 0x7efd3c0a5fb0 2023-09-30T00:53:06.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.770+0000 7efd424b8700 10 monclient(hunting): handle_auth_done global_id 4688 payload 274 2023-09-30T00:53:06.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.770+0000 7efd424b8700 1 -- 172.21.15.40:0/3656355395 >> v2:172.21.15.40:3300/0 conn(0x7efd3c0b1460 msgr2=0x7efd3c0b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:06.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.770+0000 7efd424b8700 1 --2- 172.21.15.40:0/3656355395 >> v2:172.21.15.40:3300/0 conn(0x7efd3c0b1460 0x7efd3c0b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:06.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.770+0000 7efd424b8700 1 -- 172.21.15.40:0/3656355395 >> v1:172.21.15.135:6789/0 conn(0x7efd3c0a5fb0 legacy=0x7efd3c141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:06.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.770+0000 7efd424b8700 10 monclient: _finish_hunting 0 2023-09-30T00:53:06.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.770+0000 7efd424b8700 1 monclient: found mon.noname-b 2023-09-30T00:53:06.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.770+0000 7efd424b8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:06.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.770+0000 7efd424b8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:53:06.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.770+0000 7efd424b8700 1 -- 172.21.15.40:0/3656355395 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c0a63c0 con 0x7efd3c13ac20 2023-09-30T00:53:06.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.773+0000 7efd424b8700 1 --2- 172.21.15.40:0/3656355395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd3c13ac20 0x7efd3c13b080 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efd34009e60 tx=0x7efd34009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=223ce01a0c46bbef server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:06.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd40cb5700 1 -- 172.21.15.40:0/3656355395 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3400e070 con 0x7efd3c13ac20 2023-09-30T00:53:06.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd40cb5700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:06.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd40cb5700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:06.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd40cb5700 10 monclient: dump: 2023-09-30T00:53:06.779 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:06.779 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:06.779 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:06.780 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd40cb5700 10 monclient: _finish_auth 0 2023-09-30T00:53:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd40cb5700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:06.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd40cb5700 1 -- 172.21.15.40:0/3656355395 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3400bab0 con 0x7efd3c13ac20 2023-09-30T00:53:06.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd40cb5700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:06.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd40cb5700 1 -- 172.21.15.40:0/3656355395 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3400be70 con 0x7efd3c13ac20 2023-09-30T00:53:06.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd40cb5700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:06.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd40cb5700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:53:06.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd40cb5700 10 monclient: dump: 2023-09-30T00:53:06.781 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:06.781 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:06.781 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.782 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.782 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:06.782 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:06.782 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:06.782 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:06.782 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:06.782 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:06.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd43f1b700 5 monclient: authenticate success, global_id 4688 2023-09-30T00:53:06.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd43f1b700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:06.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.777+0000 7efd43f1b700 4 set_mon_vals no callback set 2023-09-30T00:53:06.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.778+0000 7efd43f1b700 10 monclient: shutdown 2023-09-30T00:53:06.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.778+0000 7efd43f1b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:06.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.778+0000 7efd43f1b700 1 -- 172.21.15.40:0/3656355395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd3c13ac20 msgr2=0x7efd3c13b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:06.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.778+0000 7efd43f1b700 1 --2- 172.21.15.40:0/3656355395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd3c13ac20 0x7efd3c13b080 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efd34009e60 tx=0x7efd34009780 comp rx=0 tx=0).stop 2023-09-30T00:53:06.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.778+0000 7efd43f1b700 1 -- 172.21.15.40:0/3656355395 shutdown_connections 2023-09-30T00:53:06.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.778+0000 7efd43f1b700 1 --2- 172.21.15.40:0/3656355395 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd3c13ac20 0x7efd3c13b080 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:06.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.778+0000 7efd43f1b700 1 --2- 172.21.15.40:0/3656355395 >> v2:172.21.15.40:3300/0 conn(0x7efd3c0b1460 0x7efd3c0b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:06.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.778+0000 7efd43f1b700 1 -- 172.21.15.40:0/3656355395 >> 172.21.15.40:0/3656355395 conn(0x7efd3c0631f0 msgr2=0x7efd3c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:06.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.778+0000 7efd43f1b700 1 -- 172.21.15.40:0/3656355395 shutdown_connections 2023-09-30T00:53:06.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.778+0000 7efd43f1b700 1 -- 172.21.15.40:0/3656355395 wait complete. 2023-09-30T00:53:06.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.779+0000 7efd43f1b700 10 monclient: build_initial_monmap 2023-09-30T00:53:06.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.779+0000 7efd43f1b700 10 monclient: monmap: 2023-09-30T00:53:06.787 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:06.787 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:06.787 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:06.787 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:06.787 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:06.788 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:06.788 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:06.788 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:06.788 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:06.788 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:06.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.779+0000 7efd43f1b700 1 librados: starting msgr at 2023-09-30T00:53:06.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.779+0000 7efd43f1b700 1 librados: starting objecter 2023-09-30T00:53:06.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.779+0000 7efd43f1b700 1 Processor -- start 2023-09-30T00:53:06.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.779+0000 7efd43f1b700 1 -- start start 2023-09-30T00:53:06.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.779+0000 7efd43f1b700 1 librados: setting wanted keys 2023-09-30T00:53:06.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.779+0000 7efd43f1b700 1 librados: calling monclient init 2023-09-30T00:53:06.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.779+0000 7efd43f1b700 10 monclient: init 2023-09-30T00:53:06.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:06.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:53:06.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd3c0a5fb0 0x7efd3c0a93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:06.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 10 monclient(hunting): picked mon.noname-a con 0x7efd3c0a5fb0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:06.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd3c0b1460 0x7efd3c0a99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:06.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 10 monclient(hunting): picked mon.noname-c con 0x7efd3c0b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:06.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd3c13ac20 0x7efd3c150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:06.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 10 monclient(hunting): picked mon.noname-b con 0x7efd3c13ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:06.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:06.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c0aa100 con 0x7efd3c0a5fb0 2023-09-30T00:53:06.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:06.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c13f080 con 0x7efd3c13ac20 2023-09-30T00:53:06.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:06.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3c0aa590 con 0x7efd3c0b1460 2023-09-30T00:53:06.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:06.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.780+0000 7efd43f1b700 10 monclient(hunting): authenticate will time out at 5648.081723s 2023-09-30T00:53:06.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.781+0000 7efd424b8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd3c13ac20 0x7efd3c150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:06.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.781+0000 7efd414b6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd3c0b1460 0x7efd3c0a99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:06.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.781+0000 7efd41cb7700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd3c0a5fb0 0x7efd3c0a93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:06.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.781+0000 7efd424b8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd3c13ac20 0x7efd3c150c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:33594/0 (socket says 172.21.15.40:33594) 2023-09-30T00:53:06.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.781+0000 7efd424b8700 1 -- 172.21.15.40:0/3790074683 learned_addr learned my addr 172.21.15.40:0/3790074683 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:06.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.781+0000 7efd424b8700 10 monclient(hunting): get_auth_request con 0x7efd3c13ac20 auth_method 0 2023-09-30T00:53:06.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.781+0000 7efd424b8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:06.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.781+0000 7efd424b8700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:06.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.781+0000 7efd424b8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:06.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.781+0000 7efd414b6700 10 monclient(hunting): get_auth_request con 0x7efd3c0b1460 auth_method 0 2023-09-30T00:53:06.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.781+0000 7efd414b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:06.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.781+0000 7efd414b6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:06.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.781+0000 7efd414b6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:06.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.782+0000 7efd41cb7700 10 monclient(hunting): get_auth_request con 0x7efd3c0a5fb0 auth_method 0 2023-09-30T00:53:06.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.782+0000 7efd41cb7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:06.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.782+0000 7efd41cb7700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:06.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.782+0000 7efd41cb7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:06.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.782+0000 7efd424b8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:06.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.782+0000 7efd424b8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:06.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.782+0000 7efd424b8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:06.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.782+0000 7efd414b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:06.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.782+0000 7efd414b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:06.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.782+0000 7efd414b6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:06.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.782+0000 7efd41cb7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:06.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.782+0000 7efd41cb7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:06.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.782+0000 7efd41cb7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:06.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.785+0000 7efd424b8700 10 monclient(hunting): handle_auth_done global_id 4882 payload 824 2023-09-30T00:53:06.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.785+0000 7efd424b8700 1 -- 172.21.15.40:0/3790074683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd3c0b1460 msgr2=0x7efd3c0a99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:06.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.785+0000 7efd424b8700 1 --2- 172.21.15.40:0/3790074683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd3c0b1460 0x7efd3c0a99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:06.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.785+0000 7efd424b8700 1 -- 172.21.15.40:0/3790074683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd3c0a5fb0 msgr2=0x7efd3c0a93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:06.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.785+0000 7efd424b8700 1 --2- 172.21.15.40:0/3790074683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd3c0a5fb0 0x7efd3c0a93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:06.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.785+0000 7efd424b8700 10 monclient: _finish_hunting 0 2023-09-30T00:53:06.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.785+0000 7efd424b8700 1 monclient: found mon.noname-b 2023-09-30T00:53:06.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.785+0000 7efd424b8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:06.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.785+0000 7efd424b8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:06.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.785+0000 7efd424b8700 1 -- 172.21.15.40:0/3790074683 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd34009a20 con 0x7efd3c13ac20 2023-09-30T00:53:06.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.788+0000 7efd424b8700 1 --2- 172.21.15.40:0/3790074683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd3c13ac20 0x7efd3c150c10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efd340099f0 tx=0x7efd34019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:06.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.788+0000 7efd31ffb700 10 client.?.objecter ms_handle_connect 0x7efd3c13ac20 2023-09-30T00:53:06.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.788+0000 7efd31ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:06.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd31ffb700 1 -- 172.21.15.40:0/3790074683 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd34012d80 con 0x7efd3c13ac20 2023-09-30T00:53:06.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd31ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:06.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd31ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:06.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd31ffb700 10 monclient: dump: 2023-09-30T00:53:06.798 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:06.798 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:06.799 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.799 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.799 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:06.799 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:06.799 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:06.799 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:06.799 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:06.799 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:06.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd31ffb700 10 monclient: _finish_auth 0 2023-09-30T00:53:06.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd31ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:06.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd31ffb700 1 -- 172.21.15.40:0/3790074683 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd34027470 con 0x7efd3c13ac20 2023-09-30T00:53:06.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd31ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:06.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd31ffb700 1 -- 172.21.15.40:0/3790074683 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd34027830 con 0x7efd3c13ac20 2023-09-30T00:53:06.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd43f1b700 5 monclient: authenticate success, global_id 4882 2023-09-30T00:53:06.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd43f1b700 10 monclient: _renew_subs 2023-09-30T00:53:06.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd43f1b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:06.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd33fff700 4 set_mon_vals no callback set 2023-09-30T00:53:06.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd43f1b700 1 -- 172.21.15.40:0/3790074683 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3c13e750 con 0x7efd3c13ac20 2023-09-30T00:53:06.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd31ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:06.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd31ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:06.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd31ffb700 10 monclient: dump: 2023-09-30T00:53:06.800 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:06.800 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:06.800 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.801 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:06.801 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:06.801 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:06.801 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:06.801 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:06.801 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:06.801 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:06.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd43f1b700 10 client.4882.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:06.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd43f1b700 10 monclient: _renew_subs 2023-09-30T00:53:06.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd43f1b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:06.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd43f1b700 1 -- 172.21.15.40:0/3790074683 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3c151310 con 0x7efd3c13ac20 2023-09-30T00:53:06.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.792+0000 7efd43f1b700 1 librados: init done 2023-09-30T00:53:06.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.793+0000 7efd43f1b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:06.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.793+0000 7efd43f1b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:06.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.793+0000 7efd43f1b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:06.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.793+0000 7efd43f1b700 1 -- 172.21.15.40:0/3790074683 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3c0aa270 con 0x7efd3c13ac20 2023-09-30T00:53:06.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.812+0000 7efd31ffb700 1 -- 172.21.15.40:0/3790074683 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7efd34024070 con 0x7efd3c13ac20 2023-09-30T00:53:06.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.812+0000 7efd31ffb700 1 --2- 172.21.15.40:0/3790074683 >> v2:172.21.15.135:6800/116426 conn(0x7efd20068890 0x7efd2006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:06.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.813+0000 7efd41cb7700 1 --2- 172.21.15.40:0/3790074683 >> v2:172.21.15.135:6800/116426 conn(0x7efd20068890 0x7efd2006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:06.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.813+0000 7efd41cb7700 10 monclient: get_auth_request con 0x7efd20068890 auth_method 0 2023-09-30T00:53:06.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.813+0000 7efd41cb7700 1 --2- 172.21.15.40:0/3790074683 >> v2:172.21.15.135:6800/116426 conn(0x7efd20068890 0x7efd2006ad40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efd38009db0 tx=0x7efd380091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:06.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:06.813+0000 7efd31ffb700 10 client.4882.objecter ms_handle_connect 0x7efd20068890 2023-09-30T00:53:07.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.047+0000 7efd31ffb700 1 -- 172.21.15.40:0/3790074683 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7efd34058310 con 0x7efd3c13ac20 2023-09-30T00:53:07.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.047+0000 7efd31ffb700 10 client.4882.objecter ms_dispatch 0x7efd3c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:07.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.047+0000 7efd31ffb700 3 client.4882.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:07.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.047+0000 7efd31ffb700 3 client.4882.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:07.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.047+0000 7efd31ffb700 20 client.4882.objecter dump_active .. 0 homeless 2023-09-30T00:53:07.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.058+0000 7efd31ffb700 1 -- 172.21.15.40:0/3790074683 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efd34053f50 con 0x7efd3c13ac20 2023-09-30T00:53:07.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.058+0000 7efd31ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:07.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.058+0000 7efd31ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:07.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.289+0000 7efd43f1b700 1 -- 172.21.15.40:0/3790074683 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd3c064a00 con 0x7efd20068890 2023-09-30T00:53:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.292+0000 7efd31ffb700 1 -- 172.21.15.40:0/3790074683 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70274 (secure 0 0 0) 0x7efd3c064a00 con 0x7efd20068890 2023-09-30T00:53:07.293 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:53:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 10 librados: watch_flush enter 2023-09-30T00:53:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 10 librados: watch_flush exit 2023-09-30T00:53:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 20 client.4882.objecter shutdown clearing up homeless session... 2023-09-30T00:53:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 10 client.4882.objecter successfully canceled tick 2023-09-30T00:53:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 1 -- 172.21.15.40:0/3790074683 >> v2:172.21.15.135:6800/116426 conn(0x7efd20068890 msgr2=0x7efd2006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 1 --2- 172.21.15.40:0/3790074683 >> v2:172.21.15.135:6800/116426 conn(0x7efd20068890 0x7efd2006ad40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efd38009db0 tx=0x7efd380091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 10 monclient: shutdown 2023-09-30T00:53:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 1 -- 172.21.15.40:0/3790074683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd3c13ac20 msgr2=0x7efd3c150c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 1 --2- 172.21.15.40:0/3790074683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd3c13ac20 0x7efd3c150c10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efd340099f0 tx=0x7efd34019000 comp rx=0 tx=0).stop 2023-09-30T00:53:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 1 -- 172.21.15.40:0/3790074683 shutdown_connections 2023-09-30T00:53:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 1 --2- 172.21.15.40:0/3790074683 >> v2:172.21.15.135:6800/116426 conn(0x7efd20068890 0x7efd2006ad40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 1 --2- 172.21.15.40:0/3790074683 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7efd3c13ac20 0x7efd3c150c10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 1 --2- 172.21.15.40:0/3790074683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd3c0a5fb0 0x7efd3c0a93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 1 --2- 172.21.15.40:0/3790074683 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7efd3c0b1460 0x7efd3c0a99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.294+0000 7efd43f1b700 1 -- 172.21.15.40:0/3790074683 >> 172.21.15.40:0/3790074683 conn(0x7efd3c0631f0 msgr2=0x7efd3c0afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.295+0000 7efd43f1b700 1 -- 172.21.15.40:0/3790074683 shutdown_connections 2023-09-30T00:53:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.295+0000 7efd43f1b700 1 -- 172.21.15.40:0/3790074683 wait complete. 2023-09-30T00:53:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.295+0000 7efd43f1b700 1 librados: shutdown 2023-09-30T00:53:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-09-30T00:53:07.309 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2023-09-30T00:53:05.938599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43464,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931256,"statfs":{"total":773094113280,"available":773049606144,"internally_reserved":0,"allocated":2506752,"data_stored":1383352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002786"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.785504+0000","last_change":"2023-09-30T00:47:54.691712+0000","last_active":"2023-09-30T00:47:54.785504+0000","last_peered":"2023-09-30T00:47:54.785504+0000","last_clean":"2023-09-30T00:47:54.785504+0000","last_became_active":"2023-09-30T00:47:54.672127+0000","last_became_peered":"2023-09-30T00:47:54.672127+0000","last_unstale":"2023-09-30T00:47:54.785504+0000","last_undegraded":"2023-09-30T00:47:54.785504+0000","last_fullsized":"2023-09-30T00:47:54.785504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:32.704584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.160955+0000","last_change":"2023-09-30T00:47:54.749019+0000","last_active":"2023-09-30T00:47:55.160955+0000","last_peered":"2023-09-30T00:47:55.160955+0000","last_clean":"2023-09-30T00:47:55.160955+0000","last_became_active":"2023-09-30T00:47:54.702773+0000","last_became_peered":"2023-09-30T00:47:54.702773+0000","last_unstale":"2023-09-30T00:47:55.160955+0000","last_undegraded":"2023-09-30T00:47:55.160955+0000","last_fullsized":"2023-09-30T00:47:55.160955+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:07:10.081341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.809132+0000","last_change":"2023-09-30T00:47:54.157108+0000","last_active":"2023-09-30T00:47:54.809132+0000","last_peered":"2023-09-30T00:47:54.809132+0000","last_clean":"2023-09-30T00:47:54.809132+0000","last_became_active":"2023-09-30T00:47:54.139464+0000","last_became_peered":"2023-09-30T00:47:54.139464+0000","last_unstale":"2023-09-30T00:47:54.809132+0000","last_undegraded":"2023-09-30T00:47:54.809132+0000","last_fullsized":"2023-09-30T00:47:54.809132+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:42:38.117935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.795839+0000","last_change":"2023-09-30T00:47:53.993668+0000","last_active":"2023-09-30T00:47:54.795839+0000","last_peered":"2023-09-30T00:47:54.795839+0000","last_clean":"2023-09-30T00:47:54.795839+0000","last_became_active":"2023-09-30T00:47:53.951882+0000","last_became_peered":"2023-09-30T00:47:53.951882+0000","last_unstale":"2023-09-30T00:47:54.795839+0000","last_undegraded":"2023-09-30T00:47:54.795839+0000","last_fullsized":"2023-09-30T00:47:54.795839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:15:02.278001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'77","reported_seq":114,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:47:57.719324+0000","last_change":"2023-09-30T00:47:47.347565+0000","last_active":"2023-09-30T00:47:57.719324+0000","last_peered":"2023-09-30T00:47:57.719324+0000","last_clean":"2023-09-30T00:47:57.719324+0000","last_became_active":"2023-09-30T00:47:46.406306+0000","last_became_peered":"2023-09-30T00:47:46.406306+0000","last_unstale":"2023-09-30T00:47:57.719324+0000","last_undegraded":"2023-09-30T00:47:57.719324+0000","last_fullsized":"2023-09-30T00:47:57.719324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:40.867840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:40.867840+0000","last_clean_scrub_stamp":"2023-09-30T00:47:40.867840+0000","objects_scrubbed":0,"log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:17:21.219288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.780151+0000","last_change":"2023-09-30T00:47:54.729272+0000","last_active":"2023-09-30T00:47:54.780151+0000","last_peered":"2023-09-30T00:47:54.780151+0000","last_clean":"2023-09-30T00:47:54.780151+0000","last_became_active":"2023-09-30T00:47:54.710752+0000","last_became_peered":"2023-09-30T00:47:54.710752+0000","last_unstale":"2023-09-30T00:47:54.780151+0000","last_undegraded":"2023-09-30T00:47:54.780151+0000","last_fullsized":"2023-09-30T00:47:54.780151+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:59:21.956005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.239361+0000","last_change":"2023-09-30T00:47:53.773125+0000","last_active":"2023-09-30T00:47:55.239361+0000","last_peered":"2023-09-30T00:47:55.239361+0000","last_clean":"2023-09-30T00:47:55.239361+0000","last_became_active":"2023-09-30T00:47:53.761460+0000","last_became_peered":"2023-09-30T00:47:53.761460+0000","last_unstale":"2023-09-30T00:47:55.239361+0000","last_undegraded":"2023-09-30T00:47:55.239361+0000","last_fullsized":"2023-09-30T00:47:55.239361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:55.538707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.791094+0000","last_change":"2023-09-30T00:47:54.294197+0000","last_active":"2023-09-30T00:47:54.791094+0000","last_peered":"2023-09-30T00:47:54.791094+0000","last_clean":"2023-09-30T00:47:54.791094+0000","last_became_active":"2023-09-30T00:47:54.279706+0000","last_became_peered":"2023-09-30T00:47:54.279706+0000","last_unstale":"2023-09-30T00:47:54.791094+0000","last_undegraded":"2023-09-30T00:47:54.791094+0000","last_fullsized":"2023-09-30T00:47:54.791094+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:24:53.339591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.240181+0000","last_change":"2023-09-30T00:47:54.352713+0000","last_active":"2023-09-30T00:47:55.240181+0000","last_peered":"2023-09-30T00:47:55.240181+0000","last_clean":"2023-09-30T00:47:55.240181+0000","last_became_active":"2023-09-30T00:47:54.305507+0000","last_became_peered":"2023-09-30T00:47:54.305507+0000","last_unstale":"2023-09-30T00:47:55.240181+0000","last_undegraded":"2023-09-30T00:47:55.240181+0000","last_fullsized":"2023-09-30T00:47:55.240181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:21:08.873800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345988,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":13.013999999999999,"5min":12.769,"15min":13.343},"min":{"1min":7.7699999999999996,"5min":6.6319999999999997,"15min":6.6319999999999997},"max":{"1min":20.431999999999999,"5min":26.166,"15min":26.166},"last":7.4299999999999997},{"interface":"front","average":{"1min":10.711,"5min":11.778,"15min":13.131},"min":{"1min":6.8440000000000003,"5min":6.8440000000000003,"15min":6.8440000000000003},"max":{"1min":15.452,"5min":32.808,"15min":32.808},"last":9.734}]},{"osd":1,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":12.897,"5min":13.236000000000001,"15min":14.891999999999999},"min":{"1min":9.0199999999999996,"5min":7.8250000000000002,"15min":7.8250000000000002},"max":{"1min":21.581,"5min":31.925999999999998,"15min":31.925999999999998},"last":11.193},{"interface":"front","average":{"1min":15.115,"5min":16.175999999999998,"15min":17.780000000000001},"min":{"1min":9.3369999999999997,"5min":8.4629999999999992,"15min":8.4629999999999992},"max":{"1min":23.300000000000001,"5min":42.737000000000002,"15min":42.737000000000002},"last":12.452999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":14.624000000000001,"5min":16.120999999999999,"15min":17.459},"min":{"1min":8.9489999999999998,"5min":8.3469999999999995,"15min":8.3469999999999995},"max":{"1min":22.853000000000002,"5min":36.436999999999998,"15min":36.436999999999998},"last":15.381},{"interface":"front","average":{"1min":18.524999999999999,"5min":19.745999999999999,"15min":25.524999999999999},"min":{"1min":11.426,"5min":10.092000000000001,"15min":10.092000000000001},"max":{"1min":22.728999999999999,"5min":164.05600000000001,"15min":164.05600000000001},"last":16.114000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":15.416,"5min":17.084,"15min":17.538},"min":{"1min":10.166,"5min":8.9570000000000007,"15min":8.9570000000000007},"max":{"1min":21.091999999999999,"5min":36.740000000000002,"15min":36.740000000000002},"last":13.535},{"interface":"front","average":{"1min":13.753,"5min":15.188000000000001,"15min":17.27},"min":{"1min":8.1069999999999993,"5min":7.4329999999999998,"15min":7.4329999999999998},"max":{"1min":18.597999999999999,"5min":39.865000000000002,"15min":39.865000000000002},"last":21.573}]},{"osd":5,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":18.372,"5min":19.262,"15min":21.207000000000001},"min":{"1min":12.487,"5min":10.294,"15min":10.294},"max":{"1min":22.335999999999999,"5min":47.259,"15min":47.259},"last":18.552},{"interface":"front","average":{"1min":18.518000000000001,"5min":19.602,"15min":21.158000000000001},"min":{"1min":10.493,"5min":9.7059999999999995,"15min":9.7059999999999995},"max":{"1min":23.382999999999999,"5min":46.417999999999999,"15min":46.417999999999999},"last":20.602}]},{"osd":6,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":20.271000000000001,"5min":20.856999999999999,"15min":22.073},"min":{"1min":13.49,"5min":11.042999999999999,"15min":11.042999999999999},"max":{"1min":25.059999999999999,"5min":49.011000000000003,"15min":49.011000000000003},"last":14.67},{"interface":"front","average":{"1min":16.920999999999999,"5min":17.010999999999999,"15min":18.483000000000001},"min":{"1min":8.9130000000000003,"5min":8.9130000000000003,"15min":8.9130000000000003},"max":{"1min":24.98,"5min":44.369,"15min":44.369},"last":22.841999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":19.391999999999999,"15min":21.218},"min":{"1min":13.249000000000001,"5min":12.590999999999999,"15min":12.590999999999999},"max":{"1min":25.725000000000001,"5min":39.171999999999997,"15min":39.171999999999997},"last":23.925999999999998},{"interface":"front","average":{"1min":19.501999999999999,"5min":21.446999999999999,"15min":23.422000000000001},"min":{"1min":13.321999999999999,"5min":11.161,"15min":11.161},"max":{"1min":24.439,"5min":41.865000000000002,"15min":41.865000000000002},"last":16.963999999999999}]}]},{"osd":2,"up_from":20,"seq":85899345987,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":12.666,"5min":12.167,"15min":12.167},"min":{"1min":6.9450000000000003,"5min":6.6920000000000002,"15min":6.6920000000000002},"max":{"1min":20.530999999999999,"5min":22.748999999999999,"15min":22.748999999999999},"last":8.0329999999999995},{"interface":"front","average":{"1min":12.608000000000001,"5min":13.143000000000001,"15min":14.31},"min":{"1min":9.4100000000000001,"5min":6.319,"15min":6.319},"max":{"1min":21.957000000000001,"5min":31.885000000000002,"15min":31.885000000000002},"last":9.2189999999999994}]},{"osd":1,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":15.48,"5min":17.213000000000001,"15min":22.105},"min":{"1min":9.6579999999999995,"5min":8.718,"15min":8.718},"max":{"1min":21.058,"5min":125.67700000000001,"15min":125.67700000000001},"last":14.252000000000001},{"interface":"front","average":{"1min":16.795999999999999,"5min":17.684999999999999,"15min":24.204999999999998},"min":{"1min":10.210000000000001,"5min":8.0370000000000008,"15min":8.0370000000000008},"max":{"1min":24.420999999999999,"5min":156.821,"15min":156.821},"last":10.555999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":10.301,"5min":18.065999999999999,"15min":34.295999999999999},"min":{"1min":7.6210000000000004,"5min":7.6210000000000004,"15min":7.6210000000000004},"max":{"1min":14.249000000000001,"5min":513.32600000000002,"15min":513.32600000000002},"last":11.363},{"interface":"front","average":{"1min":17.056999999999999,"5min":24.167999999999999,"15min":42.537999999999997},"min":{"1min":11.009,"5min":10.308,"15min":10.308},"max":{"1min":23.140999999999998,"5min":598.48900000000003,"15min":598.48900000000003},"last":19.898}]},{"osd":4,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":14.541,"5min":16.052,"15min":17.638999999999999},"min":{"1min":9.1530000000000005,"5min":8.8859999999999992,"15min":8.8859999999999992},"max":{"1min":24.239999999999998,"5min":31.702000000000002,"15min":31.702000000000002},"last":12.669},{"interface":"front","average":{"1min":16.594000000000001,"5min":16.542000000000002,"15min":17.376999999999999},"min":{"1min":8.6750000000000007,"5min":8.6750000000000007,"15min":8.6750000000000007},"max":{"1min":23.922999999999998,"5min":37.073999999999998,"15min":37.073999999999998},"last":12.032}]},{"osd":5,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":19.826000000000001,"5min":20.279,"15min":21.254000000000001},"min":{"1min":13.452999999999999,"5min":9.1440000000000001,"15min":9.1440000000000001},"max":{"1min":25.805,"5min":38.981000000000002,"15min":38.981000000000002},"last":20.878},{"interface":"front","average":{"1min":18.370999999999999,"5min":20.248000000000001,"15min":23.968},"min":{"1min":13.335000000000001,"5min":9.8949999999999996,"15min":9.8949999999999996},"max":{"1min":24.638999999999999,"5min":74.510000000000005,"15min":74.510000000000005},"last":16.526}]},{"osd":6,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":18.266999999999999,"5min":17.795000000000002,"15min":19.167000000000002},"min":{"1min":13.561,"5min":7.6150000000000002,"15min":7.6150000000000002},"max":{"1min":24.135999999999999,"5min":40.875999999999998,"15min":40.875999999999998},"last":15.532999999999999},{"interface":"front","average":{"1min":22.277000000000001,"5min":21.561,"15min":23.045999999999999},"min":{"1min":15.621,"5min":9.5809999999999995,"15min":9.5809999999999995},"max":{"1min":26.98,"5min":42.539000000000001,"15min":42.539000000000001},"last":21.940999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":20.937000000000001,"5min":21.863,"15min":24.445},"min":{"1min":14.688000000000001,"5min":12.295999999999999,"15min":12.295999999999999},"max":{"1min":26.420999999999999,"5min":76.484999999999999,"15min":76.484999999999999},"last":17.858000000000001},{"interface":"front","average":{"1min":19.401,"5min":20.391999999999999,"15min":22.324000000000002},"min":{"1min":14.641999999999999,"5min":10.223000000000001,"15min":10.223000000000001},"max":{"1min":28.007999999999999,"5min":55.265999999999998,"15min":55.265999999999998},"last":18.835000000000001}]}]},{"osd":5,"up_from":15,"seq":64424509509,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":614400,"data_stored":471299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":10.923,"5min":10.705,"15min":11.093},"min":{"1min":7.1210000000000004,"5min":5.6130000000000004,"15min":5.6130000000000004},"max":{"1min":18.591999999999999,"5min":23.507000000000001,"15min":23.507000000000001},"last":14.978999999999999},{"interface":"front","average":{"1min":11.551,"5min":10.593999999999999,"15min":10.163},"min":{"1min":7.1470000000000002,"5min":5.2130000000000001,"15min":5.2130000000000001},"max":{"1min":20.713000000000001,"5min":27.425000000000001,"15min":27.425000000000001},"last":13.018000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.073,"5min":12.210000000000001,"15min":12.234},"min":{"1min":6.8369999999999997,"5min":6.6619999999999999,"15min":6.6619999999999999},"max":{"1min":15.596,"5min":31.512,"15min":31.512},"last":12.013},{"interface":"front","average":{"1min":16.539000000000001,"5min":19.786999999999999,"15min":26.213000000000001},"min":{"1min":11.753,"5min":10.006,"15min":10.006},"max":{"1min":21.158000000000001,"5min":232.38800000000001,"15min":232.38800000000001},"last":19.048999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":9.3290000000000006,"5min":10.186,"15min":9.9290000000000003},"min":{"1min":7.0999999999999996,"5min":6.9909999999999997,"15min":6.9909999999999997},"max":{"1min":15.125999999999999,"5min":22.271000000000001,"15min":22.271000000000001},"last":12.742000000000001},{"interface":"front","average":{"1min":16.318999999999999,"5min":16.466000000000001,"15min":17.452999999999999},"min":{"1min":12.247999999999999,"5min":8.3810000000000002,"15min":8.3810000000000002},"max":{"1min":19.457000000000001,"5min":30.27,"15min":30.27},"last":18.256}]},{"osd":3,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.276,"5min":13.022,"15min":14.42},"min":{"1min":7.1829999999999998,"5min":7.1829999999999998,"15min":7.1829999999999998},"max":{"1min":17.137,"5min":42.023000000000003,"15min":42.023000000000003},"last":8.1359999999999992},{"interface":"front","average":{"1min":14.127000000000001,"5min":14.695,"15min":16.593},"min":{"1min":9.5350000000000001,"5min":7.1029999999999998,"15min":7.1029999999999998},"max":{"1min":21.387,"5min":46.786000000000001,"15min":46.786000000000001},"last":10.852}]},{"osd":4,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":15.722,"5min":14.898,"15min":14.827999999999999},"min":{"1min":10.923,"5min":6.7380000000000004,"15min":6.7380000000000004},"max":{"1min":21.215,"5min":23.788,"15min":23.788},"last":16.582000000000001},{"interface":"front","average":{"1min":17.538,"5min":18.876000000000001,"15min":22.504000000000001},"min":{"1min":11.625,"5min":9.2300000000000004,"15min":9.2300000000000004},"max":{"1min":20.460000000000001,"5min":112.548,"15min":112.548},"last":20.884}]},{"osd":6,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":13.363,"5min":14.715999999999999,"15min":17.433},"min":{"1min":6.819,"5min":6.819,"15min":6.819},"max":{"1min":22.664000000000001,"5min":106.333,"15min":106.333},"last":17.274000000000001},{"interface":"front","average":{"1min":16.849,"5min":16.956,"15min":19.303000000000001},"min":{"1min":9.9529999999999994,"5min":8.2769999999999992,"15min":8.2769999999999992},"max":{"1min":23.702999999999999,"5min":103.483,"15min":103.483},"last":11.742000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":17.23,"5min":21.253,"15min":29.43},"min":{"1min":11.779999999999999,"5min":10.369999999999999,"15min":10.369999999999999},"max":{"1min":22.402999999999999,"5min":253.74799999999999,"15min":253.74799999999999},"last":15.756},{"interface":"front","average":{"1min":17.359999999999999,"5min":20.033999999999999,"15min":30.213999999999999},"min":{"1min":9.1869999999999994,"5min":9.0310000000000006,"15min":9.0310000000000006},"max":{"1min":21.940000000000001,"5min":328.31799999999998,"15min":328.31799999999998},"last":21.754000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509508,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":9.6129999999999995,"5min":10.558,"15min":12.276},"min":{"1min":6.4080000000000004,"5min":4.556,"15min":4.556},"max":{"1min":19.038,"5min":78.191000000000003,"15min":78.191000000000003},"last":13.297000000000001},{"interface":"front","average":{"1min":11.808,"5min":11.657,"15min":13.117000000000001},"min":{"1min":6.4189999999999996,"5min":5.5439999999999996,"15min":5.5439999999999996},"max":{"1min":21.591999999999999,"5min":73.787000000000006,"15min":73.787000000000006},"last":8.1829999999999998}]},{"osd":1,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":11.846,"5min":13.288,"15min":15.319000000000001},"min":{"1min":6.6219999999999999,"5min":6.6219999999999999,"15min":6.6219999999999999},"max":{"1min":19.167999999999999,"5min":79.558000000000007,"15min":79.558000000000007},"last":10.183},{"interface":"front","average":{"1min":13.247999999999999,"5min":14.374000000000001,"15min":16.331},"min":{"1min":7.1130000000000004,"5min":7.1130000000000004,"15min":7.1130000000000004},"max":{"1min":18.385999999999999,"5min":82.048000000000002,"15min":82.048000000000002},"last":8.5830000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":15.504,"5min":16.736999999999998,"15min":18.332000000000001},"min":{"1min":9.4640000000000004,"5min":8.2889999999999997,"15min":8.2889999999999997},"max":{"1min":20.145,"5min":98.372,"15min":98.372},"last":16.635999999999999},{"interface":"front","average":{"1min":14.375999999999999,"5min":15.202999999999999,"15min":16.692},"min":{"1min":8.0009999999999994,"5min":7.2750000000000004,"15min":7.2750000000000004},"max":{"1min":22.282,"5min":96.753,"15min":96.753},"last":14.579000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":16.495999999999999,"5min":17.460000000000001,"15min":19.975999999999999},"min":{"1min":9.0860000000000003,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":20.835000000000001,"5min":100.212,"15min":100.212},"last":14.289},{"interface":"front","average":{"1min":11.829000000000001,"5min":12.417,"15min":14.603},"min":{"1min":6.8029999999999999,"5min":6.7729999999999997,"15min":6.7729999999999997},"max":{"1min":19.068999999999999,"5min":69.975999999999999,"15min":69.975999999999999},"last":8.8279999999999994}]},{"osd":5,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":16.097000000000001,"5min":14.657999999999999,"15min":18.02},"min":{"1min":7.9880000000000004,"5min":7.3070000000000004,"15min":7.3070000000000004},"max":{"1min":20.745999999999999,"5min":129.88,"15min":129.88},"last":19.850999999999999},{"interface":"front","average":{"1min":15.327999999999999,"5min":18.414000000000001,"15min":21.858000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":22.792999999999999,"5min":101.876,"15min":101.876},"last":21.454999999999998}]},{"osd":6,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":19.172000000000001,"5min":21.908999999999999,"15min":28.599},"min":{"1min":14.031000000000001,"5min":11.853999999999999,"15min":11.853999999999999},"max":{"1min":23.716999999999999,"5min":199.161,"15min":199.161},"last":17.475999999999999},{"interface":"front","average":{"1min":13.936999999999999,"5min":17.481000000000002,"15min":24.242999999999999},"min":{"1min":9.1899999999999995,"5min":8.5060000000000002,"15min":8.5060000000000002},"max":{"1min":23.106000000000002,"5min":205.011,"15min":205.011},"last":15.991}]},{"osd":7,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":16.001000000000001,"5min":18.004999999999999,"15min":22.398},"min":{"1min":9.9570000000000007,"5min":8.2110000000000003,"15min":8.2110000000000003},"max":{"1min":22.222000000000001,"5min":140.71000000000001,"15min":140.71000000000001},"last":22.48},{"interface":"front","average":{"1min":16.390999999999998,"5min":17.071999999999999,"15min":19.221},"min":{"1min":10.544,"5min":5.6509999999999998,"15min":5.6509999999999998},"max":{"1min":23.492999999999999,"5min":101.80500000000001,"15min":101.80500000000001},"last":20.305}]}]},{"osd":6,"up_from":14,"seq":60129542213,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.535,"5min":13.951000000000001,"15min":22.998000000000001},"min":{"1min":6.5380000000000003,"5min":4.5990000000000002,"15min":4.5990000000000002},"max":{"1min":17.355,"5min":350.05700000000002,"15min":350.05700000000002},"last":7.2249999999999996},{"interface":"front","average":{"1min":9.7520000000000007,"5min":13.462999999999999,"15min":23.047999999999998},"min":{"1min":5.383,"5min":5.383,"15min":5.383},"max":{"1min":18.678000000000001,"5min":352.40899999999999,"15min":352.40899999999999},"last":9.1560000000000006}]},{"osd":1,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":12.308,"5min":18.155999999999999,"15min":33.904000000000003},"min":{"1min":6.367,"5min":5.5700000000000003,"15min":5.5700000000000003},"max":{"1min":19.513000000000002,"5min":332.67399999999998,"15min":332.67399999999998},"last":13.475},{"interface":"front","average":{"1min":12.640000000000001,"5min":19.193999999999999,"15min":37.317999999999998},"min":{"1min":7.8559999999999999,"5min":7.2629999999999999,"15min":7.2629999999999999},"max":{"1min":18.895,"5min":359.57900000000001,"15min":359.57900000000001},"last":16.504000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.337999999999999,"5min":18.093,"15min":29.870000000000001},"min":{"1min":6.907,"5min":6.593,"15min":6.593},"max":{"1min":22.527999999999999,"5min":420.69299999999998,"15min":420.69299999999998},"last":8.4580000000000002},{"interface":"front","average":{"1min":16.591999999999999,"5min":17.097000000000001,"15min":26.565000000000001},"min":{"1min":10.023,"5min":6.0990000000000002,"15min":6.0990000000000002},"max":{"1min":19.998000000000001,"5min":421.416,"15min":421.416},"last":17.954999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":13.147,"5min":14.747,"15min":17.933},"min":{"1min":7.4450000000000003,"5min":5.9710000000000001,"15min":5.9710000000000001},"max":{"1min":21.074999999999999,"5min":142.905,"15min":142.905},"last":11.849},{"interface":"front","average":{"1min":17.030999999999999,"5min":19.399999999999999,"15min":23.738},"min":{"1min":13.157,"5min":11.380000000000001,"15min":11.380000000000001},"max":{"1min":19.614000000000001,"5min":177.86199999999999,"15min":177.86199999999999},"last":19.448}]},{"osd":4,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":14.191000000000001,"5min":15.872999999999999,"15min":19.667999999999999},"min":{"1min":9.0670000000000002,"5min":7.0270000000000001,"15min":7.0270000000000001},"max":{"1min":20.523,"5min":164.02699999999999,"15min":164.02699999999999},"last":18.616},{"interface":"front","average":{"1min":17.466999999999999,"5min":19.172999999999998,"15min":23.913},"min":{"1min":10.384,"5min":6.7400000000000002,"15min":6.7400000000000002},"max":{"1min":22.350000000000001,"5min":183.18000000000001,"15min":183.18000000000001},"last":21.779}]},{"osd":5,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":15.390000000000001,"15min":19.122},"min":{"1min":7.5229999999999997,"5min":7.101,"15min":7.101},"max":{"1min":17.652000000000001,"5min":163.00700000000001,"15min":163.00700000000001},"last":15.446},{"interface":"front","average":{"1min":15.941000000000001,"5min":17.882000000000001,"15min":22.271999999999998},"min":{"1min":11.317,"5min":7.2770000000000001,"15min":7.2770000000000001},"max":{"1min":21.721,"5min":177.17400000000001,"15min":177.17400000000001},"last":20.670999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":18.608000000000001,"5min":20.460000000000001,"15min":24.350000000000001},"min":{"1min":12.318,"5min":8.8469999999999995,"15min":8.8469999999999995},"max":{"1min":21.919,"5min":179.86500000000001,"15min":179.86500000000001},"last":22.663},{"interface":"front","average":{"1min":14.218,"5min":17.603999999999999,"15min":21.802},"min":{"1min":7.4619999999999997,"5min":7.4619999999999997,"15min":7.4619999999999997},"max":{"1min":19.574000000000002,"5min":164.58099999999999,"15min":164.58099999999999},"last":16.140999999999998}]}]},{"osd":7,"up_from":15,"seq":64424509508,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":9.2799999999999994,"5min":10.220000000000001,"15min":11.925000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":17.898,"5min":52.545999999999999,"15min":52.545999999999999},"last":12.744},{"interface":"front","average":{"1min":10.803000000000001,"5min":11.276999999999999,"15min":13.035},"min":{"1min":7.5449999999999999,"5min":6.6180000000000003,"15min":6.6180000000000003},"max":{"1min":17.611000000000001,"5min":60.087000000000003,"15min":60.087000000000003},"last":10.69}]},{"osd":1,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":12.558999999999999,"5min":15.726000000000001,"15min":23.890999999999998},"min":{"1min":8.3209999999999997,"5min":5.5179999999999998,"15min":5.5179999999999998},"max":{"1min":18.693999999999999,"5min":292.69600000000003,"15min":292.69600000000003},"last":8.4469999999999992},{"interface":"front","average":{"1min":10.73,"5min":17.122,"15min":33.595999999999997},"min":{"1min":7.3499999999999996,"5min":6.9059999999999997,"15min":6.9059999999999997},"max":{"1min":19.689,"5min":571.03399999999999,"15min":571.03399999999999},"last":8.7119999999999997}]},{"osd":2,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":16.391999999999999,"5min":16.396999999999998,"15min":18.145},"min":{"1min":11.743,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":22.370000000000001,"5min":62.984000000000002,"15min":62.984000000000002},"last":15.945},{"interface":"front","average":{"1min":13.821999999999999,"5min":14.734,"15min":16.702000000000002},"min":{"1min":8.3650000000000002,"5min":7.875,"15min":7.875},"max":{"1min":21.393000000000001,"5min":59.280999999999999,"15min":59.280999999999999},"last":14.462999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":13.872999999999999,"5min":16.023,"15min":17.783000000000001},"min":{"1min":9.3239999999999998,"5min":7.8949999999999996,"15min":7.8949999999999996},"max":{"1min":21.843,"5min":65.084000000000003,"15min":65.084000000000003},"last":13.635},{"interface":"front","average":{"1min":16.149999999999999,"5min":14.801,"15min":16.591000000000001},"min":{"1min":8.1530000000000005,"5min":6.3650000000000002,"15min":6.3650000000000002},"max":{"1min":21.564,"5min":56.024999999999999,"15min":56.024999999999999},"last":17.960000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":14.815,"5min":15.505000000000001,"15min":16.719000000000001},"min":{"1min":9.6229999999999993,"5min":6.5369999999999999,"15min":6.5369999999999999},"max":{"1min":20.928000000000001,"5min":39.575000000000003,"15min":39.575000000000003},"last":15.119},{"interface":"front","average":{"1min":16.652999999999999,"5min":18.010000000000002,"15min":20.782},"min":{"1min":9.0199999999999996,"5min":8.5459999999999994,"15min":8.5459999999999994},"max":{"1min":20.765999999999998,"5min":113.193,"15min":113.193},"last":20.492000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":18.452000000000002,"5min":18.542000000000002,"15min":19.276},"min":{"1min":15.332000000000001,"5min":10.16,"15min":10.16},"max":{"1min":22.913,"5min":49.593000000000004,"15min":49.593000000000004},"last":21.762},{"interface":"front","average":{"1min":16.960999999999999,"5min":18.082000000000001,"15min":19.195},"min":{"1min":12.218999999999999,"5min":7.6820000000000004,"15min":7.6820000000000004},"max":{"1min":20.646000000000001,"5min":48.204000000000001,"15min":48.204000000000001},"last":19.838999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":18.614999999999998,"5min":19.957999999999998,"15min":23.268000000000001},"min":{"1min":14.460000000000001,"5min":10.625999999999999,"15min":10.625999999999999},"max":{"1min":23.527000000000001,"5min":115.434,"15min":115.434},"last":22.923999999999999},{"interface":"front","average":{"1min":17.231999999999999,"5min":16.93,"15min":17.43},"min":{"1min":12.894,"5min":8.0190000000000001,"15min":8.0190000000000001},"max":{"1min":22.411999999999999,"5min":40.805,"15min":40.805},"last":18.908999999999999}]}]},{"osd":0,"up_from":20,"seq":85899345988,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":12.696999999999999,"5min":13.567,"15min":14.483000000000001},"min":{"1min":5.0830000000000002,"5min":5.0830000000000002,"15min":5.0830000000000002},"max":{"1min":20.931000000000001,"5min":29.138999999999999,"15min":29.138999999999999},"last":8.5210000000000008},{"interface":"front","average":{"1min":11.182,"5min":12.558999999999999,"15min":13.718999999999999},"min":{"1min":6.9989999999999997,"5min":6.1059999999999999,"15min":6.1059999999999999},"max":{"1min":20.055,"5min":35.32,"15min":35.32},"last":12.678000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":9.7750000000000004,"5min":11.563000000000001,"15min":14.34},"min":{"1min":7.3109999999999999,"5min":6.9829999999999997,"15min":6.9829999999999997},"max":{"1min":14.356,"5min":56.628999999999998,"15min":56.628999999999998},"last":12.984},{"interface":"front","average":{"1min":16.048999999999999,"5min":17.128,"15min":19.702999999999999},"min":{"1min":11.364000000000001,"5min":10.457000000000001,"15min":10.457000000000001},"max":{"1min":20.681000000000001,"5min":50.646000000000001,"15min":50.646000000000001},"last":9.9819999999999993}]},{"osd":3,"last update":"Sat Sep 30 00:52:02 2023","interfaces":[{"interface":"back","average":{"1min":14.92,"5min":29.306999999999999,"15min":44.177},"min":{"1min":11.141,"5min":8.4399999999999995,"15min":8.4399999999999995},"max":{"1min":26.937000000000001,"5min":616.83500000000004,"15min":616.83500000000004},"last":16.058},{"interface":"front","average":{"1min":14.122999999999999,"5min":27.134,"15min":38.993000000000002},"min":{"1min":9.3360000000000003,"5min":9.1229999999999993,"15min":9.1229999999999993},"max":{"1min":22.553999999999998,"5min":495.32600000000002,"15min":495.32600000000002},"last":9.0950000000000006}]},{"osd":4,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":17.096,"5min":17.635999999999999,"15min":19.018999999999998},"min":{"1min":11.279999999999999,"5min":9.875,"15min":9.875},"max":{"1min":23.553999999999998,"5min":36.109000000000002,"15min":36.109000000000002},"last":17.013999999999999},{"interface":"front","average":{"1min":15.769,"5min":16.484999999999999,"15min":18.593},"min":{"1min":10.702,"5min":8.4420000000000002,"15min":8.4420000000000002},"max":{"1min":22.722999999999999,"5min":39.177999999999997,"15min":39.177999999999997},"last":15.233000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":18.515000000000001,"5min":19.923999999999999,"15min":22.416},"min":{"1min":12.333,"5min":9.3870000000000005,"15min":9.3870000000000005},"max":{"1min":26.995999999999999,"5min":66.313999999999993,"15min":66.313999999999993},"last":20.018000000000001},{"interface":"front","average":{"1min":20.390999999999998,"5min":22.050999999999998,"15min":23.927},"min":{"1min":12.085000000000001,"5min":12.085000000000001,"15min":12.085000000000001},"max":{"1min":25.503,"5min":41.634,"15min":41.634},"last":22.291}]},{"osd":6,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":14.702999999999999,"5min":17.117999999999999,"15min":22.646000000000001},"min":{"1min":10.412000000000001,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":21.728000000000002,"5min":54.033999999999999,"15min":54.033999999999999},"last":19.402000000000001},{"interface":"front","average":{"1min":20.824999999999999,"5min":22.52,"15min":24.59},"min":{"1min":16.77,"5min":11.055999999999999,"15min":11.055999999999999},"max":{"1min":24.806000000000001,"5min":69.688000000000002,"15min":69.688000000000002},"last":21.431999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":22.016999999999999,"5min":23.597999999999999,"15min":26.277000000000001},"min":{"1min":16.298999999999999,"5min":14.875,"15min":14.875},"max":{"1min":27.872,"5min":49.347999999999999,"15min":49.347999999999999},"last":23.643999999999998},{"interface":"front","average":{"1min":16.75,"5min":18.710999999999999,"15min":21.469000000000001},"min":{"1min":10.622,"5min":10.489000000000001,"15min":10.489000000000001},"max":{"1min":24.265000000000001,"5min":54.734999999999999,"15min":54.734999999999999},"last":14.289}]}]},{"osd":1,"up_from":19,"seq":81604378692,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6044,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365796,"statfs":{"total":96636764160,"available":96630575104,"internally_reserved":0,"allocated":614400,"data_stored":471299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":13.113,"15min":16.02},"min":{"1min":7.4809999999999999,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":13.616,"5min":91.582999999999998,"15min":91.582999999999998},"last":8.1799999999999997},{"interface":"front","average":{"1min":12.308,"5min":13.007999999999999,"15min":14.891999999999999},"min":{"1min":8.1780000000000008,"5min":7.2889999999999997,"15min":7.2889999999999997},"max":{"1min":26.073,"5min":75.853999999999999,"15min":75.853999999999999},"last":11.861000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":12.657999999999999,"5min":14.127000000000001,"15min":18.731000000000002},"min":{"1min":8.2639999999999993,"5min":8.2639999999999993,"15min":8.2639999999999993},"max":{"1min":20.702000000000002,"5min":142.41999999999999,"15min":142.41999999999999},"last":10.119999999999999},{"interface":"front","average":{"1min":15.305,"5min":16.303999999999998,"15min":19.814},"min":{"1min":9.952,"5min":8.0749999999999993,"15min":8.0749999999999993},"max":{"1min":23.823,"5min":133.988,"15min":133.988},"last":19.651}]},{"osd":3,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":14.644,"5min":17.609000000000002,"15min":23.707000000000001},"min":{"1min":8.9009999999999998,"5min":8.4019999999999992,"15min":8.4019999999999992},"max":{"1min":21.539000000000001,"5min":170.785,"15min":170.785},"last":12.593},{"interface":"front","average":{"1min":16.152000000000001,"5min":19.57,"15min":27.786000000000001},"min":{"1min":9.1349999999999998,"5min":9.1349999999999998,"15min":9.1349999999999998},"max":{"1min":23.108000000000001,"5min":170.17400000000001,"15min":170.17400000000001},"last":13.301}]},{"osd":4,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":15.271000000000001,"5min":18.404,"15min":24.379000000000001},"min":{"1min":10.272,"5min":8.9510000000000005,"15min":8.9510000000000005},"max":{"1min":19.027999999999999,"5min":108.045,"15min":108.045},"last":15.673},{"interface":"front","average":{"1min":17.634,"5min":20.315000000000001,"15min":27.440999999999999},"min":{"1min":10.829000000000001,"5min":9.9000000000000004,"15min":9.9000000000000004},"max":{"1min":23.558,"5min":141.97300000000001,"15min":141.97300000000001},"last":10.782999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":16.079000000000001,"5min":20.795999999999999,"15min":26.815999999999999},"min":{"1min":9.9489999999999998,"5min":9.9489999999999998,"15min":9.9489999999999998},"max":{"1min":22.390000000000001,"5min":116.873,"15min":116.873},"last":14.743},{"interface":"front","average":{"1min":19.495000000000001,"5min":22.710999999999999,"15min":31.870999999999999},"min":{"1min":11.393000000000001,"5min":9.9550000000000001,"15min":9.9550000000000001},"max":{"1min":24.968,"5min":199.773,"15min":199.773},"last":16.725999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":19.128,"5min":22.716000000000001,"15min":28.771999999999998},"min":{"1min":12.276999999999999,"5min":10.601000000000001,"15min":10.601000000000001},"max":{"1min":23.670999999999999,"5min":131.06299999999999,"15min":131.06299999999999},"last":17.422000000000001},{"interface":"front","average":{"1min":20.283000000000001,"5min":23.718,"15min":31.068000000000001},"min":{"1min":14.736000000000001,"5min":13.375999999999999,"15min":13.375999999999999},"max":{"1min":27.178000000000001,"5min":139.024,"15min":139.024},"last":18.157}]},{"osd":7,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":16.785,"5min":19.227,"15min":26.094999999999999},"min":{"1min":9.577,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":27.773,"5min":127.039,"15min":127.039},"last":20.626000000000001},{"interface":"front","average":{"1min":18.727,"5min":21.361999999999998,"15min":27.317},"min":{"1min":11.532,"5min":10.449,"15min":10.449},"max":{"1min":24.533999999999999,"5min":130.88499999999999,"15min":130.88499999999999},"last":21.276}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:53:07.311 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T00:53:07.311 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:53:07.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.446+0000 7f3d6e526700 10 monclient: get_monmap_and_config 2023-09-30T00:53:07.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.446+0000 7f3d6e526700 10 monclient: build_initial_monmap 2023-09-30T00:53:07.447 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.446+0000 7f3d6e526700 10 monclient: monmap: 2023-09-30T00:53:07.447 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:07.447 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:07.448 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:07.447867+0000 2023-09-30T00:53:07.448 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:07.447867+0000 2023-09-30T00:53:07.448 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:07.448 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:07.448 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:07.448 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:07.448 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:07.448 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:07.449 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:07.449 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:07.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.447+0000 7f3d6e526700 1 Processor -- start 2023-09-30T00:53:07.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.447+0000 7f3d6e526700 1 -- start start 2023-09-30T00:53:07.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.447+0000 7f3d6e526700 10 monclient: init 2023-09-30T00:53:07.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.448+0000 7f3d6e526700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:07.450 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.448+0000 7f3d6e526700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2023-09-30T00:53:07.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.448+0000 7f3d6e526700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3d680a5710 0x7f3d680a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:07.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.448+0000 7f3d6e526700 10 monclient(hunting): picked mon.noname-c con 0x7f3d680a5710 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:07.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.448+0000 7f3d6e526700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3d680a7b20 0x7f3d6813e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:07.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.448+0000 7f3d6e526700 10 monclient(hunting): picked mon.noname-a con 0x7f3d680a7b20 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:07.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.448+0000 7f3d6e526700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d6813e710 addr v1:172.21.15.40:6789/0 2023-09-30T00:53:07.451 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.448+0000 7f3d6e526700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d680a7ff0 con 0x7f3d6813e710 2023-09-30T00:53:07.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.448+0000 7f3d6e526700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:07.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.448+0000 7f3d6e526700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d680aedc0 con 0x7f3d680a7b20 2023-09-30T00:53:07.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.448+0000 7f3d6e526700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:07.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.448+0000 7f3d6e526700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d680aef00 con 0x7f3d680a5710 2023-09-30T00:53:07.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.448+0000 7f3d6e526700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:07.452 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.448+0000 7f3d6e526700 10 monclient(hunting): authenticate will time out at 5648.749515s 2023-09-30T00:53:07.459 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.459+0000 7f3d67fff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3d680a5710 0x7f3d680a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:07.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.459+0000 7f3d67fff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3d680a5710 0x7f3d680a75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:33608/0 (socket says 172.21.15.40:33608) 2023-09-30T00:53:07.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.459+0000 7f3d67fff700 1 -- 172.21.15.40:0/3203051481 learned_addr learned my addr 172.21.15.40:0/3203051481 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:07.460 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.459+0000 7f3d67fff700 10 monclient(hunting): get_auth_request con 0x7f3d680a5710 auth_method 0 2023-09-30T00:53:07.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.459+0000 7f3d67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:07.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.459+0000 7f3d67fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:07.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.459+0000 7f3d67fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:07.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.460+0000 7f3d67fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:07.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.460+0000 7f3d67fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:07.461 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.460+0000 7f3d67fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:07.462 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.462+0000 7f3d67fff700 10 monclient(hunting): handle_auth_done global_id 4885 payload 274 2023-09-30T00:53:07.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.462+0000 7f3d67fff700 1 -- 172.21.15.40:0/3203051481 >> v2:172.21.15.40:3300/0 conn(0x7f3d680a7b20 msgr2=0x7f3d6813e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:07.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.462+0000 7f3d67fff700 1 --2- 172.21.15.40:0/3203051481 >> v2:172.21.15.40:3300/0 conn(0x7f3d680a7b20 0x7f3d6813e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:07.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.462+0000 7f3d67fff700 1 -- 172.21.15.40:0/3203051481 >> v1:172.21.15.40:6789/0 conn(0x7f3d6813e710 legacy=0x7f3d68141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:07.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.462+0000 7f3d67fff700 10 monclient: _finish_hunting 0 2023-09-30T00:53:07.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.462+0000 7f3d67fff700 1 monclient: found mon.noname-c 2023-09-30T00:53:07.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.462+0000 7f3d67fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:07.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.462+0000 7f3d67fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:53:07.463 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.462+0000 7f3d67fff700 1 -- 172.21.15.40:0/3203051481 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6813ebb0 con 0x7f3d680a5710 2023-09-30T00:53:07.464 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.464+0000 7f3d67fff700 1 --2- 172.21.15.40:0/3203051481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d680a5710 0x7f3d680a75e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3d680b1d90 tx=0x7f3d58009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=457923e012614c9c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:07.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d66ffd700 1 -- 172.21.15.40:0/3203051481 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5800b820 con 0x7f3d680a5710 2023-09-30T00:53:07.468 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:07.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d66ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:53:07.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d66ffd700 10 monclient: dump: 2023-09-30T00:53:07.469 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:07.469 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:07.469 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:07.469 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:07.469 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:07.469 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:07.469 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:07.469 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:07.469 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:07.470 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:07.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d66ffd700 10 monclient: _finish_auth 0 2023-09-30T00:53:07.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d66ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:07.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d66ffd700 1 -- 172.21.15.40:0/3203051481 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5800b980 con 0x7f3d680a5710 2023-09-30T00:53:07.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d6e526700 5 monclient: authenticate success, global_id 4885 2023-09-30T00:53:07.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d6e526700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:07.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d66ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:07.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d6e526700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:07.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d66ffd700 1 -- 172.21.15.40:0/3203051481 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5800bd20 con 0x7f3d680a5710 2023-09-30T00:53:07.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d6e526700 4 set_mon_vals no callback set 2023-09-30T00:53:07.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:07.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d66ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:07.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d66ffd700 10 monclient: dump: 2023-09-30T00:53:07.471 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:07.471 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:07.471 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:07.471 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:07.471 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:07.471 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:07.471 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:07.471 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:07.471 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:07.472 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:07.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d6e526700 10 monclient: shutdown 2023-09-30T00:53:07.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.468+0000 7f3d6e526700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:07.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.469+0000 7f3d6e526700 1 -- 172.21.15.40:0/3203051481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d680a5710 msgr2=0x7f3d680a75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:07.474 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.469+0000 7f3d6e526700 1 --2- 172.21.15.40:0/3203051481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d680a5710 0x7f3d680a75e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3d680b1d90 tx=0x7f3d58009250 comp rx=0 tx=0).stop 2023-09-30T00:53:07.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.469+0000 7f3d6e526700 1 -- 172.21.15.40:0/3203051481 shutdown_connections 2023-09-30T00:53:07.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.469+0000 7f3d6e526700 1 --2- 172.21.15.40:0/3203051481 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d680a5710 0x7f3d680a75e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:07.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.469+0000 7f3d6e526700 1 --2- 172.21.15.40:0/3203051481 >> v2:172.21.15.40:3300/0 conn(0x7f3d680a7b20 0x7f3d6813e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:07.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.469+0000 7f3d6e526700 1 -- 172.21.15.40:0/3203051481 >> 172.21.15.40:0/3203051481 conn(0x7f3d68069080 msgr2=0x7f3d680631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:07.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.469+0000 7f3d6e526700 1 -- 172.21.15.40:0/3203051481 shutdown_connections 2023-09-30T00:53:07.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.469+0000 7f3d6e526700 1 -- 172.21.15.40:0/3203051481 wait complete. 2023-09-30T00:53:07.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.469+0000 7f3d6e526700 10 monclient: build_initial_monmap 2023-09-30T00:53:07.475 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.469+0000 7f3d6e526700 10 monclient: monmap: 2023-09-30T00:53:07.476 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:07.476 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:07.476 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:07.476 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:07.476 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:07.476 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:07.476 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:07.476 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:07.476 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:07.476 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:07.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.470+0000 7f3d6e526700 1 librados: starting msgr at 2023-09-30T00:53:07.476 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.470+0000 7f3d6e526700 1 librados: starting objecter 2023-09-30T00:53:07.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.470+0000 7f3d6e526700 1 Processor -- start 2023-09-30T00:53:07.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.470+0000 7f3d6e526700 1 -- start start 2023-09-30T00:53:07.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.470+0000 7f3d6e526700 1 librados: setting wanted keys 2023-09-30T00:53:07.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.470+0000 7f3d6e526700 1 librados: calling monclient init 2023-09-30T00:53:07.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.470+0000 7f3d6e526700 10 monclient: init 2023-09-30T00:53:07.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:07.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:53:07.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d680a5710 0x7f3d68141550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:07.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 10 monclient(hunting): picked mon.noname-c con 0x7f3d680a5710 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:07.477 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d680a7b20 0x7f3d6814e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:07.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 10 monclient(hunting): picked mon.noname-a con 0x7f3d680a7b20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:07.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d6813e710 0x7f3d6814e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:07.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 10 monclient(hunting): picked mon.noname-b con 0x7f3d6813e710 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:07.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:07.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d680ad410 con 0x7f3d680a7b20 2023-09-30T00:53:07.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:07.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d680abb80 con 0x7f3d6813e710 2023-09-30T00:53:07.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:07.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6814eec0 con 0x7f3d680a5710 2023-09-30T00:53:07.478 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:07.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6e526700 10 monclient(hunting): authenticate will time out at 5648.772259s 2023-09-30T00:53:07.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.471+0000 7f3d6cac3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d6813e710 0x7f3d6814e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:07.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.472+0000 7f3d6cac3700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d6813e710 0x7f3d6814e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:33614/0 (socket says 172.21.15.40:33614) 2023-09-30T00:53:07.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.472+0000 7f3d6cac3700 1 -- 172.21.15.40:0/3629802022 learned_addr learned my addr 172.21.15.40:0/3629802022 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:07.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.472+0000 7f3d6cac3700 10 monclient(hunting): get_auth_request con 0x7f3d6813e710 auth_method 0 2023-09-30T00:53:07.479 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.472+0000 7f3d6cac3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:07.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.472+0000 7f3d6cac3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:07.480 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.472+0000 7f3d6cac3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:07.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.472+0000 7f3d6cac3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:07.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.472+0000 7f3d6cac3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:07.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.472+0000 7f3d6cac3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:07.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.473+0000 7f3d677fe700 1 --2- 172.21.15.40:0/3629802022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d680a7b20 0x7f3d6814e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:07.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.473+0000 7f3d677fe700 10 monclient(hunting): get_auth_request con 0x7f3d680a7b20 auth_method 0 2023-09-30T00:53:07.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.473+0000 7f3d677fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:07.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.473+0000 7f3d677fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:07.481 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.473+0000 7f3d677fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:07.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.474+0000 7f3d67fff700 1 --2- 172.21.15.40:0/3629802022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d680a5710 0x7f3d68141550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:07.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.474+0000 7f3d677fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:07.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.474+0000 7f3d677fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:07.483 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d677fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:07.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d67fff700 10 monclient(hunting): get_auth_request con 0x7f3d680a5710 auth_method 0 2023-09-30T00:53:07.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d67fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:07.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d67fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:07.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d67fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:07.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d6cac3700 10 monclient(hunting): handle_auth_done global_id 4888 payload 824 2023-09-30T00:53:07.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d6cac3700 1 -- 172.21.15.40:0/3629802022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d680a5710 msgr2=0x7f3d68141550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:07.484 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d6cac3700 1 --2- 172.21.15.40:0/3629802022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d680a5710 0x7f3d68141550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:07.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d6cac3700 1 -- 172.21.15.40:0/3629802022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d680a7b20 msgr2=0x7f3d6814e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:07.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d6cac3700 1 --2- 172.21.15.40:0/3629802022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d680a7b20 0x7f3d6814e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:07.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d6cac3700 10 monclient: _finish_hunting 0 2023-09-30T00:53:07.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d6cac3700 1 monclient: found mon.noname-b 2023-09-30T00:53:07.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d6cac3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:07.485 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d6cac3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:07.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.475+0000 7f3d6cac3700 1 -- 172.21.15.40:0/3629802022 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d580094f0 con 0x7f3d6813e710 2023-09-30T00:53:07.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.478+0000 7f3d6cac3700 1 --2- 172.21.15.40:0/3629802022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d6813e710 0x7f3d6814e980 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000bbb0 tx=0x7f3d60009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:07.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.478+0000 7f3d4ffff700 10 client.?.objecter ms_handle_connect 0x7f3d6813e710 2023-09-30T00:53:07.486 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.478+0000 7f3d4ffff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:07.488 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d4ffff700 1 -- 172.21.15.40:0/3629802022 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d600092e0 con 0x7f3d6813e710 2023-09-30T00:53:07.488 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:07.488 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d4ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:07.489 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d4ffff700 10 monclient: dump: 2023-09-30T00:53:07.489 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:07.489 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:07.489 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:07.489 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:07.489 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:07.490 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:07.490 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:07.490 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:07.490 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:07.490 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:07.490 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d4ffff700 10 monclient: _finish_auth 0 2023-09-30T00:53:07.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d4ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:07.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d4ffff700 1 -- 172.21.15.40:0/3629802022 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6000c6f0 con 0x7f3d6813e710 2023-09-30T00:53:07.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d4ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:07.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d4ffff700 1 -- 172.21.15.40:0/3629802022 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6000ca90 con 0x7f3d6813e710 2023-09-30T00:53:07.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d4ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:07.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d4ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:07.491 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d4ffff700 10 monclient: dump: 2023-09-30T00:53:07.492 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:07.492 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:07.492 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:07.492 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:07.492 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:07.492 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:07.492 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:07.492 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:07.493 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:07.493 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:07.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d6e526700 5 monclient: authenticate success, global_id 4888 2023-09-30T00:53:07.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d6e526700 10 monclient: _renew_subs 2023-09-30T00:53:07.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d6e526700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:07.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d6e526700 1 -- 172.21.15.40:0/3629802022 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d680ac070 con 0x7f3d6813e710 2023-09-30T00:53:07.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d667fc700 4 set_mon_vals no callback set 2023-09-30T00:53:07.493 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d6e526700 10 client.4888.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:07.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d6e526700 10 monclient: _renew_subs 2023-09-30T00:53:07.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d6e526700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:07.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d6e526700 1 -- 172.21.15.40:0/3629802022 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d680ab1e0 con 0x7f3d6813e710 2023-09-30T00:53:07.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.484+0000 7f3d6e526700 1 librados: init done 2023-09-30T00:53:07.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.485+0000 7f3d6e526700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:07.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.485+0000 7f3d6e526700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:07.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.485+0000 7f3d6e526700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:07.494 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.485+0000 7f3d6e526700 1 -- 172.21.15.40:0/3629802022 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d68072b90 con 0x7f3d6813e710 2023-09-30T00:53:07.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.503+0000 7f3d4ffff700 1 -- 172.21.15.40:0/3629802022 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3d60012070 con 0x7f3d6813e710 2023-09-30T00:53:07.504 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.504+0000 7f3d4ffff700 1 --2- 172.21.15.40:0/3629802022 >> v2:172.21.15.135:6800/116426 conn(0x7f3d54068890 0x7f3d5406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.504+0000 7f3d67fff700 1 --2- 172.21.15.40:0/3629802022 >> v2:172.21.15.135:6800/116426 conn(0x7f3d54068890 0x7f3d5406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.504+0000 7f3d67fff700 10 monclient: get_auth_request con 0x7f3d54068890 auth_method 0 2023-09-30T00:53:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.505+0000 7f3d4ffff700 10 client.4888.objecter ms_handle_connect 0x7f3d54068890 2023-09-30T00:53:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.505+0000 7f3d67fff700 1 --2- 172.21.15.40:0/3629802022 >> v2:172.21.15.135:6800/116426 conn(0x7f3d54068890 0x7f3d5406ad40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3d680b1d90 tx=0x7f3d58016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:07.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.736+0000 7f3d4ffff700 1 -- 172.21.15.40:0/3629802022 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3d60055ab0 con 0x7f3d6813e710 2023-09-30T00:53:07.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.736+0000 7f3d4ffff700 10 client.4888.objecter ms_dispatch 0x7f3d68001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:07.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.736+0000 7f3d4ffff700 3 client.4888.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:07.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.736+0000 7f3d4ffff700 3 client.4888.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:07.737 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.736+0000 7f3d4ffff700 20 client.4888.objecter dump_active .. 0 homeless 2023-09-30T00:53:07.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.747+0000 7f3d4ffff700 1 -- 172.21.15.40:0/3629802022 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3d600516f0 con 0x7f3d6813e710 2023-09-30T00:53:07.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.747+0000 7f3d4ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:07.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.747+0000 7f3d4ffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:07.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.951+0000 7f3d6e526700 1 -- 172.21.15.40:0/3629802022 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3d680abcc0 con 0x7f3d54068890 2023-09-30T00:53:07.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.955+0000 7f3d4ffff700 1 -- 172.21.15.40:0/3629802022 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70250 (secure 0 0 0) 0x7f3d680abcc0 con 0x7f3d54068890 2023-09-30T00:53:07.956 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:53:07.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 10 librados: watch_flush enter 2023-09-30T00:53:07.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 10 librados: watch_flush exit 2023-09-30T00:53:07.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 20 client.4888.objecter shutdown clearing up homeless session... 2023-09-30T00:53:07.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 10 client.4888.objecter successfully canceled tick 2023-09-30T00:53:07.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 1 -- 172.21.15.40:0/3629802022 >> v2:172.21.15.135:6800/116426 conn(0x7f3d54068890 msgr2=0x7f3d5406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:07.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 1 --2- 172.21.15.40:0/3629802022 >> v2:172.21.15.135:6800/116426 conn(0x7f3d54068890 0x7f3d5406ad40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3d680b1d90 tx=0x7f3d58016000 comp rx=0 tx=0).stop 2023-09-30T00:53:07.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 10 monclient: shutdown 2023-09-30T00:53:07.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:07.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 1 -- 172.21.15.40:0/3629802022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d6813e710 msgr2=0x7f3d6814e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:07.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 1 --2- 172.21.15.40:0/3629802022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d6813e710 0x7f3d6814e980 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000bbb0 tx=0x7f3d60009840 comp rx=0 tx=0).stop 2023-09-30T00:53:07.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 1 -- 172.21.15.40:0/3629802022 shutdown_connections 2023-09-30T00:53:07.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 1 --2- 172.21.15.40:0/3629802022 >> v2:172.21.15.135:6800/116426 conn(0x7f3d54068890 0x7f3d5406ad40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:07.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 1 --2- 172.21.15.40:0/3629802022 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3d6813e710 0x7f3d6814e980 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:07.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 1 --2- 172.21.15.40:0/3629802022 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3d680a5710 0x7f3d68141550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:07.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 1 --2- 172.21.15.40:0/3629802022 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3d680a7b20 0x7f3d6814e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:07.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 1 -- 172.21.15.40:0/3629802022 >> 172.21.15.40:0/3629802022 conn(0x7f3d68069080 msgr2=0x7f3d6813a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:07.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 1 -- 172.21.15.40:0/3629802022 shutdown_connections 2023-09-30T00:53:07.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 1 -- 172.21.15.40:0/3629802022 wait complete. 2023-09-30T00:53:07.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:07.957+0000 7f3d6e526700 1 librados: shutdown 2023-09-30T00:53:07.960 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-09-30T00:53:07.971 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2023-09-30T00:53:07.939000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43464,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931256,"statfs":{"total":773094113280,"available":773049606144,"internally_reserved":0,"allocated":2506752,"data_stored":1383352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002685"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.785504+0000","last_change":"2023-09-30T00:47:54.691712+0000","last_active":"2023-09-30T00:47:54.785504+0000","last_peered":"2023-09-30T00:47:54.785504+0000","last_clean":"2023-09-30T00:47:54.785504+0000","last_became_active":"2023-09-30T00:47:54.672127+0000","last_became_peered":"2023-09-30T00:47:54.672127+0000","last_unstale":"2023-09-30T00:47:54.785504+0000","last_undegraded":"2023-09-30T00:47:54.785504+0000","last_fullsized":"2023-09-30T00:47:54.785504+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:15:32.704584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.160955+0000","last_change":"2023-09-30T00:47:54.749019+0000","last_active":"2023-09-30T00:47:55.160955+0000","last_peered":"2023-09-30T00:47:55.160955+0000","last_clean":"2023-09-30T00:47:55.160955+0000","last_became_active":"2023-09-30T00:47:54.702773+0000","last_became_peered":"2023-09-30T00:47:54.702773+0000","last_unstale":"2023-09-30T00:47:55.160955+0000","last_undegraded":"2023-09-30T00:47:55.160955+0000","last_fullsized":"2023-09-30T00:47:55.160955+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:07:10.081341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.809132+0000","last_change":"2023-09-30T00:47:54.157108+0000","last_active":"2023-09-30T00:47:54.809132+0000","last_peered":"2023-09-30T00:47:54.809132+0000","last_clean":"2023-09-30T00:47:54.809132+0000","last_became_active":"2023-09-30T00:47:54.139464+0000","last_became_peered":"2023-09-30T00:47:54.139464+0000","last_unstale":"2023-09-30T00:47:54.809132+0000","last_undegraded":"2023-09-30T00:47:54.809132+0000","last_fullsized":"2023-09-30T00:47:54.809132+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:42:38.117935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.795839+0000","last_change":"2023-09-30T00:47:53.993668+0000","last_active":"2023-09-30T00:47:54.795839+0000","last_peered":"2023-09-30T00:47:54.795839+0000","last_clean":"2023-09-30T00:47:54.795839+0000","last_became_active":"2023-09-30T00:47:53.951882+0000","last_became_peered":"2023-09-30T00:47:53.951882+0000","last_unstale":"2023-09-30T00:47:54.795839+0000","last_undegraded":"2023-09-30T00:47:54.795839+0000","last_fullsized":"2023-09-30T00:47:54.795839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:15:02.278001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'77","reported_seq":114,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:47:57.719324+0000","last_change":"2023-09-30T00:47:47.347565+0000","last_active":"2023-09-30T00:47:57.719324+0000","last_peered":"2023-09-30T00:47:57.719324+0000","last_clean":"2023-09-30T00:47:57.719324+0000","last_became_active":"2023-09-30T00:47:46.406306+0000","last_became_peered":"2023-09-30T00:47:46.406306+0000","last_unstale":"2023-09-30T00:47:57.719324+0000","last_undegraded":"2023-09-30T00:47:57.719324+0000","last_fullsized":"2023-09-30T00:47:57.719324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:40.867840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:40.867840+0000","last_clean_scrub_stamp":"2023-09-30T00:47:40.867840+0000","objects_scrubbed":0,"log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:17:21.219288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.780151+0000","last_change":"2023-09-30T00:47:54.729272+0000","last_active":"2023-09-30T00:47:54.780151+0000","last_peered":"2023-09-30T00:47:54.780151+0000","last_clean":"2023-09-30T00:47:54.780151+0000","last_became_active":"2023-09-30T00:47:54.710752+0000","last_became_peered":"2023-09-30T00:47:54.710752+0000","last_unstale":"2023-09-30T00:47:54.780151+0000","last_undegraded":"2023-09-30T00:47:54.780151+0000","last_fullsized":"2023-09-30T00:47:54.780151+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:59:21.956005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.239361+0000","last_change":"2023-09-30T00:47:53.773125+0000","last_active":"2023-09-30T00:47:55.239361+0000","last_peered":"2023-09-30T00:47:55.239361+0000","last_clean":"2023-09-30T00:47:55.239361+0000","last_became_active":"2023-09-30T00:47:53.761460+0000","last_became_peered":"2023-09-30T00:47:53.761460+0000","last_unstale":"2023-09-30T00:47:55.239361+0000","last_undegraded":"2023-09-30T00:47:55.239361+0000","last_fullsized":"2023-09-30T00:47:55.239361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:55.538707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:54.791094+0000","last_change":"2023-09-30T00:47:54.294197+0000","last_active":"2023-09-30T00:47:54.791094+0000","last_peered":"2023-09-30T00:47:54.791094+0000","last_clean":"2023-09-30T00:47:54.791094+0000","last_became_active":"2023-09-30T00:47:54.279706+0000","last_became_peered":"2023-09-30T00:47:54.279706+0000","last_unstale":"2023-09-30T00:47:54.791094+0000","last_undegraded":"2023-09-30T00:47:54.791094+0000","last_fullsized":"2023-09-30T00:47:54.791094+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:24:53.339591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.240181+0000","last_change":"2023-09-30T00:47:54.352713+0000","last_active":"2023-09-30T00:47:55.240181+0000","last_peered":"2023-09-30T00:47:55.240181+0000","last_clean":"2023-09-30T00:47:55.240181+0000","last_became_active":"2023-09-30T00:47:54.305507+0000","last_became_peered":"2023-09-30T00:47:54.305507+0000","last_unstale":"2023-09-30T00:47:55.240181+0000","last_undegraded":"2023-09-30T00:47:55.240181+0000","last_fullsized":"2023-09-30T00:47:55.240181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:21:08.873800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345989,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":13.013999999999999,"5min":12.769,"15min":13.343},"min":{"1min":7.7699999999999996,"5min":6.6319999999999997,"15min":6.6319999999999997},"max":{"1min":20.431999999999999,"5min":26.166,"15min":26.166},"last":19.902000000000001},{"interface":"front","average":{"1min":10.711,"5min":11.778,"15min":13.131},"min":{"1min":6.8440000000000003,"5min":6.8440000000000003,"15min":6.8440000000000003},"max":{"1min":15.452,"5min":32.808,"15min":32.808},"last":9.4649999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":12.897,"5min":13.236000000000001,"15min":14.891999999999999},"min":{"1min":9.0199999999999996,"5min":7.8250000000000002,"15min":7.8250000000000002},"max":{"1min":21.581,"5min":31.925999999999998,"15min":31.925999999999998},"last":11.721},{"interface":"front","average":{"1min":15.115,"5min":16.175999999999998,"15min":17.780000000000001},"min":{"1min":9.3369999999999997,"5min":8.4629999999999992,"15min":8.4629999999999992},"max":{"1min":23.300000000000001,"5min":42.737000000000002,"15min":42.737000000000002},"last":21.065999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":14.624000000000001,"5min":16.120999999999999,"15min":17.459},"min":{"1min":8.9489999999999998,"5min":8.3469999999999995,"15min":8.3469999999999995},"max":{"1min":22.853000000000002,"5min":36.436999999999998,"15min":36.436999999999998},"last":13.814},{"interface":"front","average":{"1min":18.524999999999999,"5min":19.745999999999999,"15min":25.524999999999999},"min":{"1min":11.426,"5min":10.092000000000001,"15min":10.092000000000001},"max":{"1min":22.728999999999999,"5min":164.05600000000001,"15min":164.05600000000001},"last":22.655999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":15.416,"5min":17.084,"15min":17.538},"min":{"1min":10.166,"5min":8.9570000000000007,"15min":8.9570000000000007},"max":{"1min":21.091999999999999,"5min":36.740000000000002,"15min":36.740000000000002},"last":17.593},{"interface":"front","average":{"1min":13.753,"5min":15.188000000000001,"15min":17.27},"min":{"1min":8.1069999999999993,"5min":7.4329999999999998,"15min":7.4329999999999998},"max":{"1min":18.597999999999999,"5min":39.865000000000002,"15min":39.865000000000002},"last":15.675000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":18.372,"5min":19.262,"15min":21.207000000000001},"min":{"1min":12.487,"5min":10.294,"15min":10.294},"max":{"1min":22.335999999999999,"5min":47.259,"15min":47.259},"last":16.344999999999999},{"interface":"front","average":{"1min":18.518000000000001,"5min":19.602,"15min":21.158000000000001},"min":{"1min":10.493,"5min":9.7059999999999995,"15min":9.7059999999999995},"max":{"1min":23.382999999999999,"5min":46.417999999999999,"15min":46.417999999999999},"last":24.184000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":20.271000000000001,"5min":20.856999999999999,"15min":22.073},"min":{"1min":13.49,"5min":11.042999999999999,"15min":11.042999999999999},"max":{"1min":25.059999999999999,"5min":49.011000000000003,"15min":49.011000000000003},"last":25.294},{"interface":"front","average":{"1min":16.920999999999999,"5min":17.010999999999999,"15min":18.483000000000001},"min":{"1min":8.9130000000000003,"5min":8.9130000000000003,"15min":8.9130000000000003},"max":{"1min":24.98,"5min":44.369,"15min":44.369},"last":11.391}]},{"osd":7,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":19.391999999999999,"15min":21.218},"min":{"1min":13.249000000000001,"5min":12.590999999999999,"15min":12.590999999999999},"max":{"1min":25.725000000000001,"5min":39.171999999999997,"15min":39.171999999999997},"last":14.52},{"interface":"front","average":{"1min":19.501999999999999,"5min":21.446999999999999,"15min":23.422000000000001},"min":{"1min":13.321999999999999,"5min":11.161,"15min":11.161},"max":{"1min":24.439,"5min":41.865000000000002,"15min":41.865000000000002},"last":18.565999999999999}]}]},{"osd":2,"up_from":20,"seq":85899345988,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":12.666,"5min":12.167,"15min":12.167},"min":{"1min":6.9450000000000003,"5min":6.6920000000000002,"15min":6.6920000000000002},"max":{"1min":20.530999999999999,"5min":22.748999999999999,"15min":22.748999999999999},"last":10.007},{"interface":"front","average":{"1min":12.608000000000001,"5min":13.143000000000001,"15min":14.31},"min":{"1min":9.4100000000000001,"5min":6.319,"15min":6.319},"max":{"1min":21.957000000000001,"5min":31.885000000000002,"15min":31.885000000000002},"last":12.263}]},{"osd":1,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":15.48,"5min":17.213000000000001,"15min":22.105},"min":{"1min":9.6579999999999995,"5min":8.718,"15min":8.718},"max":{"1min":21.058,"5min":125.67700000000001,"15min":125.67700000000001},"last":16.972999999999999},{"interface":"front","average":{"1min":16.795999999999999,"5min":17.684999999999999,"15min":24.204999999999998},"min":{"1min":10.210000000000001,"5min":8.0370000000000008,"15min":8.0370000000000008},"max":{"1min":24.420999999999999,"5min":156.821,"15min":156.821},"last":10.519}]},{"osd":3,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":10.301,"5min":18.065999999999999,"15min":34.295999999999999},"min":{"1min":7.6210000000000004,"5min":7.6210000000000004,"15min":7.6210000000000004},"max":{"1min":14.249000000000001,"5min":513.32600000000002,"15min":513.32600000000002},"last":9.6359999999999992},{"interface":"front","average":{"1min":17.056999999999999,"5min":24.167999999999999,"15min":42.537999999999997},"min":{"1min":11.009,"5min":10.308,"15min":10.308},"max":{"1min":23.140999999999998,"5min":598.48900000000003,"15min":598.48900000000003},"last":20.390000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":14.541,"5min":16.052,"15min":17.638999999999999},"min":{"1min":9.1530000000000005,"5min":8.8859999999999992,"15min":8.8859999999999992},"max":{"1min":24.239999999999998,"5min":31.702000000000002,"15min":31.702000000000002},"last":12.999000000000001},{"interface":"front","average":{"1min":16.594000000000001,"5min":16.542000000000002,"15min":17.376999999999999},"min":{"1min":8.6750000000000007,"5min":8.6750000000000007,"15min":8.6750000000000007},"max":{"1min":23.922999999999998,"5min":37.073999999999998,"15min":37.073999999999998},"last":18.332999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":19.826000000000001,"5min":20.279,"15min":21.254000000000001},"min":{"1min":13.452999999999999,"5min":9.1440000000000001,"15min":9.1440000000000001},"max":{"1min":25.805,"5min":38.981000000000002,"15min":38.981000000000002},"last":21.93},{"interface":"front","average":{"1min":18.370999999999999,"5min":20.248000000000001,"15min":23.968},"min":{"1min":13.335000000000001,"5min":9.8949999999999996,"15min":9.8949999999999996},"max":{"1min":24.638999999999999,"5min":74.510000000000005,"15min":74.510000000000005},"last":22.577000000000002}]},{"osd":6,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":18.266999999999999,"5min":17.795000000000002,"15min":19.167000000000002},"min":{"1min":13.561,"5min":7.6150000000000002,"15min":7.6150000000000002},"max":{"1min":24.135999999999999,"5min":40.875999999999998,"15min":40.875999999999998},"last":15.689},{"interface":"front","average":{"1min":22.277000000000001,"5min":21.561,"15min":23.045999999999999},"min":{"1min":15.621,"5min":9.5809999999999995,"15min":9.5809999999999995},"max":{"1min":26.98,"5min":42.539000000000001,"15min":42.539000000000001},"last":24.202000000000002}]},{"osd":7,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":20.937000000000001,"5min":21.863,"15min":24.445},"min":{"1min":14.688000000000001,"5min":12.295999999999999,"15min":12.295999999999999},"max":{"1min":26.420999999999999,"5min":76.484999999999999,"15min":76.484999999999999},"last":21.058},{"interface":"front","average":{"1min":19.401,"5min":20.391999999999999,"15min":22.324000000000002},"min":{"1min":14.641999999999999,"5min":10.223000000000001,"15min":10.223000000000001},"max":{"1min":28.007999999999999,"5min":55.265999999999998,"15min":55.265999999999998},"last":16.696999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509509,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":614400,"data_stored":471299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":10.923,"5min":10.705,"15min":11.093},"min":{"1min":7.1210000000000004,"5min":5.6130000000000004,"15min":5.6130000000000004},"max":{"1min":18.591999999999999,"5min":23.507000000000001,"15min":23.507000000000001},"last":14.978999999999999},{"interface":"front","average":{"1min":11.551,"5min":10.593999999999999,"15min":10.163},"min":{"1min":7.1470000000000002,"5min":5.2130000000000001,"15min":5.2130000000000001},"max":{"1min":20.713000000000001,"5min":27.425000000000001,"15min":27.425000000000001},"last":13.018000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.073,"5min":12.210000000000001,"15min":12.234},"min":{"1min":6.8369999999999997,"5min":6.6619999999999999,"15min":6.6619999999999999},"max":{"1min":15.596,"5min":31.512,"15min":31.512},"last":12.013},{"interface":"front","average":{"1min":16.539000000000001,"5min":19.786999999999999,"15min":26.213000000000001},"min":{"1min":11.753,"5min":10.006,"15min":10.006},"max":{"1min":21.158000000000001,"5min":232.38800000000001,"15min":232.38800000000001},"last":19.048999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":9.3290000000000006,"5min":10.186,"15min":9.9290000000000003},"min":{"1min":7.0999999999999996,"5min":6.9909999999999997,"15min":6.9909999999999997},"max":{"1min":15.125999999999999,"5min":22.271000000000001,"15min":22.271000000000001},"last":12.742000000000001},{"interface":"front","average":{"1min":16.318999999999999,"5min":16.466000000000001,"15min":17.452999999999999},"min":{"1min":12.247999999999999,"5min":8.3810000000000002,"15min":8.3810000000000002},"max":{"1min":19.457000000000001,"5min":30.27,"15min":30.27},"last":18.256}]},{"osd":3,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.276,"5min":13.022,"15min":14.42},"min":{"1min":7.1829999999999998,"5min":7.1829999999999998,"15min":7.1829999999999998},"max":{"1min":17.137,"5min":42.023000000000003,"15min":42.023000000000003},"last":8.1359999999999992},{"interface":"front","average":{"1min":14.127000000000001,"5min":14.695,"15min":16.593},"min":{"1min":9.5350000000000001,"5min":7.1029999999999998,"15min":7.1029999999999998},"max":{"1min":21.387,"5min":46.786000000000001,"15min":46.786000000000001},"last":10.852}]},{"osd":4,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":15.722,"5min":14.898,"15min":14.827999999999999},"min":{"1min":10.923,"5min":6.7380000000000004,"15min":6.7380000000000004},"max":{"1min":21.215,"5min":23.788,"15min":23.788},"last":16.582000000000001},{"interface":"front","average":{"1min":17.538,"5min":18.876000000000001,"15min":22.504000000000001},"min":{"1min":11.625,"5min":9.2300000000000004,"15min":9.2300000000000004},"max":{"1min":20.460000000000001,"5min":112.548,"15min":112.548},"last":20.884}]},{"osd":6,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":13.363,"5min":14.715999999999999,"15min":17.433},"min":{"1min":6.819,"5min":6.819,"15min":6.819},"max":{"1min":22.664000000000001,"5min":106.333,"15min":106.333},"last":17.274000000000001},{"interface":"front","average":{"1min":16.849,"5min":16.956,"15min":19.303000000000001},"min":{"1min":9.9529999999999994,"5min":8.2769999999999992,"15min":8.2769999999999992},"max":{"1min":23.702999999999999,"5min":103.483,"15min":103.483},"last":11.742000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":17.23,"5min":21.253,"15min":29.43},"min":{"1min":11.779999999999999,"5min":10.369999999999999,"15min":10.369999999999999},"max":{"1min":22.402999999999999,"5min":253.74799999999999,"15min":253.74799999999999},"last":15.756},{"interface":"front","average":{"1min":17.359999999999999,"5min":20.033999999999999,"15min":30.213999999999999},"min":{"1min":9.1869999999999994,"5min":9.0310000000000006,"15min":9.0310000000000006},"max":{"1min":21.940000000000001,"5min":328.31799999999998,"15min":328.31799999999998},"last":21.754000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509508,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":9.6129999999999995,"5min":10.558,"15min":12.276},"min":{"1min":6.4080000000000004,"5min":4.556,"15min":4.556},"max":{"1min":19.038,"5min":78.191000000000003,"15min":78.191000000000003},"last":13.297000000000001},{"interface":"front","average":{"1min":11.808,"5min":11.657,"15min":13.117000000000001},"min":{"1min":6.4189999999999996,"5min":5.5439999999999996,"15min":5.5439999999999996},"max":{"1min":21.591999999999999,"5min":73.787000000000006,"15min":73.787000000000006},"last":8.1829999999999998}]},{"osd":1,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":11.846,"5min":13.288,"15min":15.319000000000001},"min":{"1min":6.6219999999999999,"5min":6.6219999999999999,"15min":6.6219999999999999},"max":{"1min":19.167999999999999,"5min":79.558000000000007,"15min":79.558000000000007},"last":10.183},{"interface":"front","average":{"1min":13.247999999999999,"5min":14.374000000000001,"15min":16.331},"min":{"1min":7.1130000000000004,"5min":7.1130000000000004,"15min":7.1130000000000004},"max":{"1min":18.385999999999999,"5min":82.048000000000002,"15min":82.048000000000002},"last":8.5830000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":15.504,"5min":16.736999999999998,"15min":18.332000000000001},"min":{"1min":9.4640000000000004,"5min":8.2889999999999997,"15min":8.2889999999999997},"max":{"1min":20.145,"5min":98.372,"15min":98.372},"last":16.635999999999999},{"interface":"front","average":{"1min":14.375999999999999,"5min":15.202999999999999,"15min":16.692},"min":{"1min":8.0009999999999994,"5min":7.2750000000000004,"15min":7.2750000000000004},"max":{"1min":22.282,"5min":96.753,"15min":96.753},"last":14.579000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":16.495999999999999,"5min":17.460000000000001,"15min":19.975999999999999},"min":{"1min":9.0860000000000003,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":20.835000000000001,"5min":100.212,"15min":100.212},"last":14.289},{"interface":"front","average":{"1min":11.829000000000001,"5min":12.417,"15min":14.603},"min":{"1min":6.8029999999999999,"5min":6.7729999999999997,"15min":6.7729999999999997},"max":{"1min":19.068999999999999,"5min":69.975999999999999,"15min":69.975999999999999},"last":8.8279999999999994}]},{"osd":5,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":16.097000000000001,"5min":14.657999999999999,"15min":18.02},"min":{"1min":7.9880000000000004,"5min":7.3070000000000004,"15min":7.3070000000000004},"max":{"1min":20.745999999999999,"5min":129.88,"15min":129.88},"last":19.850999999999999},{"interface":"front","average":{"1min":15.327999999999999,"5min":18.414000000000001,"15min":21.858000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":22.792999999999999,"5min":101.876,"15min":101.876},"last":21.454999999999998}]},{"osd":6,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":19.172000000000001,"5min":21.908999999999999,"15min":28.599},"min":{"1min":14.031000000000001,"5min":11.853999999999999,"15min":11.853999999999999},"max":{"1min":23.716999999999999,"5min":199.161,"15min":199.161},"last":17.475999999999999},{"interface":"front","average":{"1min":13.936999999999999,"5min":17.481000000000002,"15min":24.242999999999999},"min":{"1min":9.1899999999999995,"5min":8.5060000000000002,"15min":8.5060000000000002},"max":{"1min":23.106000000000002,"5min":205.011,"15min":205.011},"last":15.991}]},{"osd":7,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":16.001000000000001,"5min":18.004999999999999,"15min":22.398},"min":{"1min":9.9570000000000007,"5min":8.2110000000000003,"15min":8.2110000000000003},"max":{"1min":22.222000000000001,"5min":140.71000000000001,"15min":140.71000000000001},"last":22.48},{"interface":"front","average":{"1min":16.390999999999998,"5min":17.071999999999999,"15min":19.221},"min":{"1min":10.544,"5min":5.6509999999999998,"15min":5.6509999999999998},"max":{"1min":23.492999999999999,"5min":101.80500000000001,"15min":101.80500000000001},"last":20.305}]}]},{"osd":6,"up_from":14,"seq":60129542213,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.535,"5min":13.951000000000001,"15min":22.998000000000001},"min":{"1min":6.5380000000000003,"5min":4.5990000000000002,"15min":4.5990000000000002},"max":{"1min":17.355,"5min":350.05700000000002,"15min":350.05700000000002},"last":7.2249999999999996},{"interface":"front","average":{"1min":9.7520000000000007,"5min":13.462999999999999,"15min":23.047999999999998},"min":{"1min":5.383,"5min":5.383,"15min":5.383},"max":{"1min":18.678000000000001,"5min":352.40899999999999,"15min":352.40899999999999},"last":9.1560000000000006}]},{"osd":1,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":12.308,"5min":18.155999999999999,"15min":33.904000000000003},"min":{"1min":6.367,"5min":5.5700000000000003,"15min":5.5700000000000003},"max":{"1min":19.513000000000002,"5min":332.67399999999998,"15min":332.67399999999998},"last":13.475},{"interface":"front","average":{"1min":12.640000000000001,"5min":19.193999999999999,"15min":37.317999999999998},"min":{"1min":7.8559999999999999,"5min":7.2629999999999999,"15min":7.2629999999999999},"max":{"1min":18.895,"5min":359.57900000000001,"15min":359.57900000000001},"last":16.504000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.337999999999999,"5min":18.093,"15min":29.870000000000001},"min":{"1min":6.907,"5min":6.593,"15min":6.593},"max":{"1min":22.527999999999999,"5min":420.69299999999998,"15min":420.69299999999998},"last":8.4580000000000002},{"interface":"front","average":{"1min":16.591999999999999,"5min":17.097000000000001,"15min":26.565000000000001},"min":{"1min":10.023,"5min":6.0990000000000002,"15min":6.0990000000000002},"max":{"1min":19.998000000000001,"5min":421.416,"15min":421.416},"last":17.954999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":13.147,"5min":14.747,"15min":17.933},"min":{"1min":7.4450000000000003,"5min":5.9710000000000001,"15min":5.9710000000000001},"max":{"1min":21.074999999999999,"5min":142.905,"15min":142.905},"last":11.849},{"interface":"front","average":{"1min":17.030999999999999,"5min":19.399999999999999,"15min":23.738},"min":{"1min":13.157,"5min":11.380000000000001,"15min":11.380000000000001},"max":{"1min":19.614000000000001,"5min":177.86199999999999,"15min":177.86199999999999},"last":19.448}]},{"osd":4,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":14.191000000000001,"5min":15.872999999999999,"15min":19.667999999999999},"min":{"1min":9.0670000000000002,"5min":7.0270000000000001,"15min":7.0270000000000001},"max":{"1min":20.523,"5min":164.02699999999999,"15min":164.02699999999999},"last":18.616},{"interface":"front","average":{"1min":17.466999999999999,"5min":19.172999999999998,"15min":23.913},"min":{"1min":10.384,"5min":6.7400000000000002,"15min":6.7400000000000002},"max":{"1min":22.350000000000001,"5min":183.18000000000001,"15min":183.18000000000001},"last":21.779}]},{"osd":5,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":15.390000000000001,"15min":19.122},"min":{"1min":7.5229999999999997,"5min":7.101,"15min":7.101},"max":{"1min":17.652000000000001,"5min":163.00700000000001,"15min":163.00700000000001},"last":15.446},{"interface":"front","average":{"1min":15.941000000000001,"5min":17.882000000000001,"15min":22.271999999999998},"min":{"1min":11.317,"5min":7.2770000000000001,"15min":7.2770000000000001},"max":{"1min":21.721,"5min":177.17400000000001,"15min":177.17400000000001},"last":20.670999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":18.608000000000001,"5min":20.460000000000001,"15min":24.350000000000001},"min":{"1min":12.318,"5min":8.8469999999999995,"15min":8.8469999999999995},"max":{"1min":21.919,"5min":179.86500000000001,"15min":179.86500000000001},"last":22.663},{"interface":"front","average":{"1min":14.218,"5min":17.603999999999999,"15min":21.802},"min":{"1min":7.4619999999999997,"5min":7.4619999999999997,"15min":7.4619999999999997},"max":{"1min":19.574000000000002,"5min":164.58099999999999,"15min":164.58099999999999},"last":16.140999999999998}]}]},{"osd":7,"up_from":15,"seq":64424509508,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":9.2799999999999994,"5min":10.220000000000001,"15min":11.925000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":17.898,"5min":52.545999999999999,"15min":52.545999999999999},"last":12.744},{"interface":"front","average":{"1min":10.803000000000001,"5min":11.276999999999999,"15min":13.035},"min":{"1min":7.5449999999999999,"5min":6.6180000000000003,"15min":6.6180000000000003},"max":{"1min":17.611000000000001,"5min":60.087000000000003,"15min":60.087000000000003},"last":10.69}]},{"osd":1,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":12.558999999999999,"5min":15.726000000000001,"15min":23.890999999999998},"min":{"1min":8.3209999999999997,"5min":5.5179999999999998,"15min":5.5179999999999998},"max":{"1min":18.693999999999999,"5min":292.69600000000003,"15min":292.69600000000003},"last":8.4469999999999992},{"interface":"front","average":{"1min":10.73,"5min":17.122,"15min":33.595999999999997},"min":{"1min":7.3499999999999996,"5min":6.9059999999999997,"15min":6.9059999999999997},"max":{"1min":19.689,"5min":571.03399999999999,"15min":571.03399999999999},"last":8.7119999999999997}]},{"osd":2,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":16.391999999999999,"5min":16.396999999999998,"15min":18.145},"min":{"1min":11.743,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":22.370000000000001,"5min":62.984000000000002,"15min":62.984000000000002},"last":15.945},{"interface":"front","average":{"1min":13.821999999999999,"5min":14.734,"15min":16.702000000000002},"min":{"1min":8.3650000000000002,"5min":7.875,"15min":7.875},"max":{"1min":21.393000000000001,"5min":59.280999999999999,"15min":59.280999999999999},"last":14.462999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":13.872999999999999,"5min":16.023,"15min":17.783000000000001},"min":{"1min":9.3239999999999998,"5min":7.8949999999999996,"15min":7.8949999999999996},"max":{"1min":21.843,"5min":65.084000000000003,"15min":65.084000000000003},"last":13.635},{"interface":"front","average":{"1min":16.149999999999999,"5min":14.801,"15min":16.591000000000001},"min":{"1min":8.1530000000000005,"5min":6.3650000000000002,"15min":6.3650000000000002},"max":{"1min":21.564,"5min":56.024999999999999,"15min":56.024999999999999},"last":17.960000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":14.815,"5min":15.505000000000001,"15min":16.719000000000001},"min":{"1min":9.6229999999999993,"5min":6.5369999999999999,"15min":6.5369999999999999},"max":{"1min":20.928000000000001,"5min":39.575000000000003,"15min":39.575000000000003},"last":15.119},{"interface":"front","average":{"1min":16.652999999999999,"5min":18.010000000000002,"15min":20.782},"min":{"1min":9.0199999999999996,"5min":8.5459999999999994,"15min":8.5459999999999994},"max":{"1min":20.765999999999998,"5min":113.193,"15min":113.193},"last":20.492000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":18.452000000000002,"5min":18.542000000000002,"15min":19.276},"min":{"1min":15.332000000000001,"5min":10.16,"15min":10.16},"max":{"1min":22.913,"5min":49.593000000000004,"15min":49.593000000000004},"last":21.762},{"interface":"front","average":{"1min":16.960999999999999,"5min":18.082000000000001,"15min":19.195},"min":{"1min":12.218999999999999,"5min":7.6820000000000004,"15min":7.6820000000000004},"max":{"1min":20.646000000000001,"5min":48.204000000000001,"15min":48.204000000000001},"last":19.838999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":18.614999999999998,"5min":19.957999999999998,"15min":23.268000000000001},"min":{"1min":14.460000000000001,"5min":10.625999999999999,"15min":10.625999999999999},"max":{"1min":23.527000000000001,"5min":115.434,"15min":115.434},"last":22.923999999999999},{"interface":"front","average":{"1min":17.231999999999999,"5min":16.93,"15min":17.43},"min":{"1min":12.894,"5min":8.0190000000000001,"15min":8.0190000000000001},"max":{"1min":22.411999999999999,"5min":40.805,"15min":40.805},"last":18.908999999999999}]}]},{"osd":0,"up_from":20,"seq":85899345989,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":12.696999999999999,"5min":13.567,"15min":14.483000000000001},"min":{"1min":5.0830000000000002,"5min":5.0830000000000002,"15min":5.0830000000000002},"max":{"1min":20.931000000000001,"5min":29.138999999999999,"15min":29.138999999999999},"last":24.564},{"interface":"front","average":{"1min":11.182,"5min":12.558999999999999,"15min":13.718999999999999},"min":{"1min":6.9989999999999997,"5min":6.1059999999999999,"15min":6.1059999999999999},"max":{"1min":20.055,"5min":35.32,"15min":35.32},"last":16.326000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":9.7750000000000004,"5min":11.563000000000001,"15min":14.34},"min":{"1min":7.3109999999999999,"5min":6.9829999999999997,"15min":6.9829999999999997},"max":{"1min":14.356,"5min":56.628999999999998,"15min":56.628999999999998},"last":11.722},{"interface":"front","average":{"1min":16.048999999999999,"5min":17.128,"15min":19.702999999999999},"min":{"1min":11.364000000000001,"5min":10.457000000000001,"15min":10.457000000000001},"max":{"1min":20.681000000000001,"5min":50.646000000000001,"15min":50.646000000000001},"last":22.198}]},{"osd":3,"last update":"Sat Sep 30 00:53:06 2023","interfaces":[{"interface":"back","average":{"1min":14.436,"5min":21.872,"15min":41.698},"min":{"1min":9.6940000000000008,"5min":8.4399999999999995,"15min":8.4399999999999995},"max":{"1min":20.186,"5min":616.83500000000004,"15min":616.83500000000004},"last":19.805},{"interface":"front","average":{"1min":14.864000000000001,"5min":21.122,"15min":36.988999999999997},"min":{"1min":8.2059999999999995,"5min":8.2059999999999995,"15min":8.2059999999999995},"max":{"1min":20.460000000000001,"5min":495.32600000000002,"15min":495.32600000000002},"last":20.460000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":17.096,"5min":17.635999999999999,"15min":19.018999999999998},"min":{"1min":11.279999999999999,"5min":9.875,"15min":9.875},"max":{"1min":23.553999999999998,"5min":36.109000000000002,"15min":36.109000000000002},"last":15.17},{"interface":"front","average":{"1min":15.769,"5min":16.484999999999999,"15min":18.593},"min":{"1min":10.702,"5min":8.4420000000000002,"15min":8.4420000000000002},"max":{"1min":22.722999999999999,"5min":39.177999999999997,"15min":39.177999999999997},"last":11.428000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":18.515000000000001,"5min":19.923999999999999,"15min":22.416},"min":{"1min":12.333,"5min":9.3870000000000005,"15min":9.3870000000000005},"max":{"1min":26.995999999999999,"5min":66.313999999999993,"15min":66.313999999999993},"last":10.776},{"interface":"front","average":{"1min":20.390999999999998,"5min":22.050999999999998,"15min":23.927},"min":{"1min":12.085000000000001,"5min":12.085000000000001,"15min":12.085000000000001},"max":{"1min":25.503,"5min":41.634,"15min":41.634},"last":25.838000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":14.702999999999999,"5min":17.117999999999999,"15min":22.646000000000001},"min":{"1min":10.412000000000001,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":21.728000000000002,"5min":54.033999999999999,"15min":54.033999999999999},"last":14.375},{"interface":"front","average":{"1min":20.824999999999999,"5min":22.52,"15min":24.59},"min":{"1min":16.77,"5min":11.055999999999999,"15min":11.055999999999999},"max":{"1min":24.806000000000001,"5min":69.688000000000002,"15min":69.688000000000002},"last":17.568999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":22.016999999999999,"5min":23.597999999999999,"15min":26.277000000000001},"min":{"1min":16.298999999999999,"5min":14.875,"15min":14.875},"max":{"1min":27.872,"5min":49.347999999999999,"15min":49.347999999999999},"last":21.228000000000002},{"interface":"front","average":{"1min":16.75,"5min":18.710999999999999,"15min":21.469000000000001},"min":{"1min":10.622,"5min":10.489000000000001,"15min":10.489000000000001},"max":{"1min":24.265000000000001,"5min":54.734999999999999,"15min":54.734999999999999},"last":18.324000000000002}]}]},{"osd":1,"up_from":19,"seq":81604378692,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6044,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365796,"statfs":{"total":96636764160,"available":96630575104,"internally_reserved":0,"allocated":614400,"data_stored":471299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":13.113,"15min":16.02},"min":{"1min":7.4809999999999999,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":13.616,"5min":91.582999999999998,"15min":91.582999999999998},"last":8.1799999999999997},{"interface":"front","average":{"1min":12.308,"5min":13.007999999999999,"15min":14.891999999999999},"min":{"1min":8.1780000000000008,"5min":7.2889999999999997,"15min":7.2889999999999997},"max":{"1min":26.073,"5min":75.853999999999999,"15min":75.853999999999999},"last":11.861000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":12.657999999999999,"5min":14.127000000000001,"15min":18.731000000000002},"min":{"1min":8.2639999999999993,"5min":8.2639999999999993,"15min":8.2639999999999993},"max":{"1min":20.702000000000002,"5min":142.41999999999999,"15min":142.41999999999999},"last":10.119999999999999},{"interface":"front","average":{"1min":15.305,"5min":16.303999999999998,"15min":19.814},"min":{"1min":9.952,"5min":8.0749999999999993,"15min":8.0749999999999993},"max":{"1min":23.823,"5min":133.988,"15min":133.988},"last":19.651}]},{"osd":3,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":14.644,"5min":17.609000000000002,"15min":23.707000000000001},"min":{"1min":8.9009999999999998,"5min":8.4019999999999992,"15min":8.4019999999999992},"max":{"1min":21.539000000000001,"5min":170.785,"15min":170.785},"last":12.593},{"interface":"front","average":{"1min":16.152000000000001,"5min":19.57,"15min":27.786000000000001},"min":{"1min":9.1349999999999998,"5min":9.1349999999999998,"15min":9.1349999999999998},"max":{"1min":23.108000000000001,"5min":170.17400000000001,"15min":170.17400000000001},"last":13.301}]},{"osd":4,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":15.271000000000001,"5min":18.404,"15min":24.379000000000001},"min":{"1min":10.272,"5min":8.9510000000000005,"15min":8.9510000000000005},"max":{"1min":19.027999999999999,"5min":108.045,"15min":108.045},"last":15.673},{"interface":"front","average":{"1min":17.634,"5min":20.315000000000001,"15min":27.440999999999999},"min":{"1min":10.829000000000001,"5min":9.9000000000000004,"15min":9.9000000000000004},"max":{"1min":23.558,"5min":141.97300000000001,"15min":141.97300000000001},"last":10.782999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":16.079000000000001,"5min":20.795999999999999,"15min":26.815999999999999},"min":{"1min":9.9489999999999998,"5min":9.9489999999999998,"15min":9.9489999999999998},"max":{"1min":22.390000000000001,"5min":116.873,"15min":116.873},"last":14.743},{"interface":"front","average":{"1min":19.495000000000001,"5min":22.710999999999999,"15min":31.870999999999999},"min":{"1min":11.393000000000001,"5min":9.9550000000000001,"15min":9.9550000000000001},"max":{"1min":24.968,"5min":199.773,"15min":199.773},"last":16.725999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":19.128,"5min":22.716000000000001,"15min":28.771999999999998},"min":{"1min":12.276999999999999,"5min":10.601000000000001,"15min":10.601000000000001},"max":{"1min":23.670999999999999,"5min":131.06299999999999,"15min":131.06299999999999},"last":17.422000000000001},{"interface":"front","average":{"1min":20.283000000000001,"5min":23.718,"15min":31.068000000000001},"min":{"1min":14.736000000000001,"5min":13.375999999999999,"15min":13.375999999999999},"max":{"1min":27.178000000000001,"5min":139.024,"15min":139.024},"last":18.157}]},{"osd":7,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":16.785,"5min":19.227,"15min":26.094999999999999},"min":{"1min":9.577,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":27.773,"5min":127.039,"15min":127.039},"last":20.626000000000001},{"interface":"front","average":{"1min":18.727,"5min":21.361999999999998,"15min":27.317},"min":{"1min":11.532,"5min":10.449,"15min":10.449},"max":{"1min":24.533999999999999,"5min":130.88499999999999,"15min":130.88499999999999},"last":21.276}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:53:07.974 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:53:08.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.106+0000 7f04f1ab5700 10 monclient: get_monmap_and_config 2023-09-30T00:53:08.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.106+0000 7f04f1ab5700 10 monclient: build_initial_monmap 2023-09-30T00:53:08.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.106+0000 7f04f1ab5700 10 monclient: monmap: 2023-09-30T00:53:08.107 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:08.107 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:08.107 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:08.107183+0000 2023-09-30T00:53:08.107 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:08.107183+0000 2023-09-30T00:53:08.107 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:08.107 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:08.107 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:08.107 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:08.107 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:08.107 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:08.107 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:08.108 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:08.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.106+0000 7f04f1ab5700 1 Processor -- start 2023-09-30T00:53:08.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.106+0000 7f04f1ab5700 1 -- start start 2023-09-30T00:53:08.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 10 monclient: init 2023-09-30T00:53:08.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:08.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-09-30T00:53:08.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f04ec0b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:08.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f04ec0b1830 0x7f04ec0a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:08.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 10 monclient(hunting): picked mon.noname-a con 0x7f04ec0b1830 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:08.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f04ec13ac20 0x7f04ec13b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:08.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 10 monclient(hunting): picked mon.noname-c con 0x7f04ec13ac20 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:08.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ec0a5b20 con 0x7f04ec0b1460 2023-09-30T00:53:08.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:08.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ec141e40 con 0x7f04ec0b1830 2023-09-30T00:53:08.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ec0a64f0 con 0x7f04ec13ac20 2023-09-30T00:53:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.107+0000 7f04f1ab5700 10 monclient(hunting): authenticate will time out at 5649.408688s 2023-09-30T00:53:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.108+0000 7f04eaffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f04ec0b1830 0x7f04ec0a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.108+0000 7f04ebfff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f04ec13ac20 0x7f04ec13b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.108+0000 7f04eaffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f04ec0b1830 0x7f04ec0a5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49966/0 (socket says 172.21.15.40:49966) 2023-09-30T00:53:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.108+0000 7f04eaffd700 1 -- 172.21.15.40:0/188516362 learned_addr learned my addr 172.21.15.40:0/188516362 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.108+0000 7f04eaffd700 10 monclient(hunting): get_auth_request con 0x7f04ec0b1830 auth_method 0 2023-09-30T00:53:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.108+0000 7f04eaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.108+0000 7f04eaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.108+0000 7f04eaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.108+0000 7f04ebfff700 10 monclient(hunting): get_auth_request con 0x7f04ec13ac20 auth_method 0 2023-09-30T00:53:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.108+0000 7f04ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.108+0000 7f04ebfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.108+0000 7f04ebfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.109+0000 7f04eaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.109+0000 7f04eaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.109+0000 7f04eaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.109+0000 7f04ebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.109+0000 7f04ebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.109+0000 7f04ebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.111+0000 7f04eaffd700 10 monclient(hunting): handle_auth_done global_id 4803 payload 274 2023-09-30T00:53:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.111+0000 7f04eaffd700 1 -- 172.21.15.40:0/188516362 >> v2:172.21.15.135:3300/0 conn(0x7f04ec13ac20 msgr2=0x7f04ec13b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.111+0000 7f04eaffd700 1 --2- 172.21.15.40:0/188516362 >> v2:172.21.15.135:3300/0 conn(0x7f04ec13ac20 0x7f04ec13b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:08.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.111+0000 7f04eaffd700 1 -- 172.21.15.40:0/188516362 >> v1:172.21.15.135:6789/0 conn(0x7f04ec0b1460 legacy=0x7f04ec141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:08.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.111+0000 7f04eaffd700 10 monclient: _finish_hunting 0 2023-09-30T00:53:08.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.111+0000 7f04eaffd700 1 monclient: found mon.noname-a 2023-09-30T00:53:08.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.111+0000 7f04eaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:08.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.111+0000 7f04eaffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:53:08.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.111+0000 7f04eaffd700 1 -- 172.21.15.40:0/188516362 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec0a71f0 con 0x7f04ec0b1830 2023-09-30T00:53:08.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.113+0000 7f04eaffd700 1 --2- 172.21.15.40:0/188516362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04ec0b1830 0x7f04ec0a5fb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f04e00099e0 tx=0x7f04e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a3460c860aaeffd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.116+0000 7f04ea7fc700 1 -- 172.21.15.40:0/188516362 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e000b7e0 con 0x7f04ec0b1830 2023-09-30T00:53:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.116+0000 7f04ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.116+0000 7f04ea7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:53:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.116+0000 7f04ea7fc700 10 monclient: dump: 2023-09-30T00:53:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:08.116 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:08.117 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.116+0000 7f04ea7fc700 10 monclient: _finish_auth 0 2023-09-30T00:53:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.116+0000 7f04ea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.116+0000 7f04f1ab5700 5 monclient: authenticate success, global_id 4803 2023-09-30T00:53:08.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.116+0000 7f04f1ab5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:08.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.117+0000 7f04ea7fc700 1 -- 172.21.15.40:0/188516362 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e000b940 con 0x7f04ec0b1830 2023-09-30T00:53:08.118 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.117+0000 7f04ea7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.117+0000 7f04ea7fc700 1 -- 172.21.15.40:0/188516362 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e000bce0 con 0x7f04ec0b1830 2023-09-30T00:53:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.117+0000 7f04ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.117+0000 7f04ea7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:53:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.117+0000 7f04ea7fc700 10 monclient: dump: 2023-09-30T00:53:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:08.120 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.117+0000 7f04f1ab5700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.117+0000 7f04f1ab5700 4 set_mon_vals no callback set 2023-09-30T00:53:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.118+0000 7f04f1ab5700 10 monclient: shutdown 2023-09-30T00:53:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.118+0000 7f04f1ab5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.118+0000 7f04f1ab5700 1 -- 172.21.15.40:0/188516362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04ec0b1830 msgr2=0x7f04ec0a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.118+0000 7f04f1ab5700 1 --2- 172.21.15.40:0/188516362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04ec0b1830 0x7f04ec0a5fb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f04e00099e0 tx=0x7f04e0009250 comp rx=0 tx=0).stop 2023-09-30T00:53:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.118+0000 7f04f1ab5700 1 -- 172.21.15.40:0/188516362 shutdown_connections 2023-09-30T00:53:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.118+0000 7f04f1ab5700 1 --2- 172.21.15.40:0/188516362 >> v2:172.21.15.135:3300/0 conn(0x7f04ec13ac20 0x7f04ec13b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.118+0000 7f04f1ab5700 1 --2- 172.21.15.40:0/188516362 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04ec0b1830 0x7f04ec0a5fb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.118+0000 7f04f1ab5700 1 -- 172.21.15.40:0/188516362 >> 172.21.15.40:0/188516362 conn(0x7f04ec0631f0 msgr2=0x7f04ec0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.118+0000 7f04f1ab5700 1 -- 172.21.15.40:0/188516362 shutdown_connections 2023-09-30T00:53:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.118+0000 7f04f1ab5700 1 -- 172.21.15.40:0/188516362 wait complete. 2023-09-30T00:53:08.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.119+0000 7f04f1ab5700 10 monclient: build_initial_monmap 2023-09-30T00:53:08.124 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.119+0000 7f04f1ab5700 10 monclient: monmap: 2023-09-30T00:53:08.124 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:08.124 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:08.124 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:08.124 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:08.124 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:08.124 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:08.124 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:08.124 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:08.125 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.119+0000 7f04f1ab5700 1 librados: starting msgr at 2023-09-30T00:53:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.119+0000 7f04f1ab5700 1 librados: starting objecter 2023-09-30T00:53:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.119+0000 7f04f1ab5700 1 Processor -- start 2023-09-30T00:53:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.119+0000 7f04f1ab5700 1 -- start start 2023-09-30T00:53:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.119+0000 7f04f1ab5700 1 librados: setting wanted keys 2023-09-30T00:53:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.119+0000 7f04f1ab5700 1 librados: calling monclient init 2023-09-30T00:53:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.119+0000 7f04f1ab5700 10 monclient: init 2023-09-30T00:53:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:53:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04ec0b1460 0x7f04ec13e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:08.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 10 monclient(hunting): picked mon.noname-c con 0x7f04ec0b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:08.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04ec0b1830 0x7f04ec1516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:08.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 10 monclient(hunting): picked mon.noname-a con 0x7f04ec0b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:08.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04ec13ac20 0x7f04ec0a9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:08.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 10 monclient(hunting): picked mon.noname-b con 0x7f04ec13ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:08.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:08.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec0ab9a0 con 0x7f04ec0b1830 2023-09-30T00:53:08.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:08.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec151c10 con 0x7f04ec13ac20 2023-09-30T00:53:08.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:08.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec151d80 con 0x7f04ec0b1460 2023-09-30T00:53:08.126 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.120+0000 7f04f1ab5700 10 monclient(hunting): authenticate will time out at 5649.421582s 2023-09-30T00:53:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04eaffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04ec0b1830 0x7f04ec1516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04eb7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04ec0b1460 0x7f04ec13e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04ebfff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04ec13ac20 0x7f04ec0a9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04eaffd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04ec0b1830 0x7f04ec1516d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49976/0 (socket says 172.21.15.40:49976) 2023-09-30T00:53:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04eaffd700 1 -- 172.21.15.40:0/1915538876 learned_addr learned my addr 172.21.15.40:0/1915538876 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04eaffd700 10 monclient(hunting): get_auth_request con 0x7f04ec0b1830 auth_method 0 2023-09-30T00:53:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04eaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04eaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04eaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04eb7fe700 10 monclient(hunting): get_auth_request con 0x7f04ec0b1460 auth_method 0 2023-09-30T00:53:08.127 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04eb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04eb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04eb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04ebfff700 10 monclient(hunting): get_auth_request con 0x7f04ec13ac20 auth_method 0 2023-09-30T00:53:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04ebfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.121+0000 7f04ebfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.122+0000 7f04eaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.122+0000 7f04eaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.122+0000 7f04eaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.122+0000 7f04ebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.122+0000 7f04ebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:08.128 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.122+0000 7f04ebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:08.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.122+0000 7f04eb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:08.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.122+0000 7f04eb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:08.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.122+0000 7f04eb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:08.129 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.125+0000 7f04ebfff700 10 monclient(hunting): handle_auth_done global_id 4897 payload 824 2023-09-30T00:53:08.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.125+0000 7f04ebfff700 1 -- 172.21.15.40:0/1915538876 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04ec0b1460 msgr2=0x7f04ec13e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:08.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.125+0000 7f04ebfff700 1 --2- 172.21.15.40:0/1915538876 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04ec0b1460 0x7f04ec13e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:08.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.125+0000 7f04ebfff700 1 -- 172.21.15.40:0/1915538876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04ec0b1830 msgr2=0x7f04ec1516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:08.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.125+0000 7f04ebfff700 1 --2- 172.21.15.40:0/1915538876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04ec0b1830 0x7f04ec1516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:08.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.125+0000 7f04ebfff700 10 monclient: _finish_hunting 0 2023-09-30T00:53:08.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.125+0000 7f04ebfff700 1 monclient: found mon.noname-b 2023-09-30T00:53:08.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.125+0000 7f04ebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:08.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.125+0000 7f04ebfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:08.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.125+0000 7f04ebfff700 1 -- 172.21.15.40:0/1915538876 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e00094f0 con 0x7f04ec13ac20 2023-09-30T00:53:08.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.127+0000 7f04ebfff700 1 --2- 172.21.15.40:0/1915538876 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04ec13ac20 0x7f04ec0a9620 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f04e400ecc0 tx=0x7f04e400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:08.130 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.127+0000 7f04d37fe700 10 client.?.objecter ms_handle_connect 0x7f04ec13ac20 2023-09-30T00:53:08.131 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.127+0000 7f04d37fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:08.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04d37fe700 1 -- 172.21.15.40:0/1915538876 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e400c8c0 con 0x7f04ec13ac20 2023-09-30T00:53:08.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:08.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04d37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:08.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04d37fe700 10 monclient: dump: 2023-09-30T00:53:08.132 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:08.132 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:08.133 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04d37fe700 10 monclient: _finish_auth 0 2023-09-30T00:53:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04d37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:08.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04d37fe700 1 -- 172.21.15.40:0/1915538876 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e400f040 con 0x7f04ec13ac20 2023-09-30T00:53:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04f1ab5700 5 monclient: authenticate success, global_id 4897 2023-09-30T00:53:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04f1ab5700 10 monclient: _renew_subs 2023-09-30T00:53:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04f1ab5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04f1ab5700 1 -- 172.21.15.40:0/1915538876 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ec0a9b60 con 0x7f04ec13ac20 2023-09-30T00:53:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04d37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04d37fe700 1 -- 172.21.15.40:0/1915538876 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e400cc90 con 0x7f04ec13ac20 2023-09-30T00:53:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04d37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04d37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04d37fe700 10 monclient: dump: 2023-09-30T00:53:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:08.134 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:08.135 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04e9ffb700 4 set_mon_vals no callback set 2023-09-30T00:53:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04f1ab5700 10 client.4897.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04f1ab5700 10 monclient: _renew_subs 2023-09-30T00:53:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04f1ab5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:08.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04f1ab5700 1 -- 172.21.15.40:0/1915538876 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ec0a9fd0 con 0x7f04ec13ac20 2023-09-30T00:53:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.131+0000 7f04f1ab5700 1 librados: init done 2023-09-30T00:53:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.132+0000 7f04f1ab5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.132+0000 7f04f1ab5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.132+0000 7f04f1ab5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:08.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.132+0000 7f04f1ab5700 1 -- 172.21.15.40:0/1915538876 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ec0a9340 con 0x7f04ec13ac20 2023-09-30T00:53:08.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.383+0000 7f04d37fe700 1 -- 172.21.15.40:0/1915538876 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f04e4014690 con 0x7f04ec13ac20 2023-09-30T00:53:08.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.384+0000 7f04d37fe700 1 --2- 172.21.15.40:0/1915538876 >> v2:172.21.15.135:6800/116426 conn(0x7f04d8068960 0x7f04d806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:08.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.384+0000 7f04eb7fe700 1 --2- 172.21.15.40:0/1915538876 >> v2:172.21.15.135:6800/116426 conn(0x7f04d8068960 0x7f04d806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:08.384 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.384+0000 7f04eb7fe700 10 monclient: get_auth_request con 0x7f04d8068960 auth_method 0 2023-09-30T00:53:08.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.385+0000 7f04eb7fe700 1 --2- 172.21.15.40:0/1915538876 >> v2:172.21.15.135:6800/116426 conn(0x7f04d8068960 0x7f04d806ae10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f04dc003a80 tx=0x7f04dc007ec0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:08.385 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.385+0000 7f04d37fe700 10 client.4897.objecter ms_handle_connect 0x7f04d8068960 2023-09-30T00:53:08.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.389+0000 7f04d37fe700 1 -- 172.21.15.40:0/1915538876 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f04e4012340 con 0x7f04ec13ac20 2023-09-30T00:53:08.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.389+0000 7f04d37fe700 10 client.4897.objecter ms_dispatch 0x7f04ec001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:08.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.389+0000 7f04d37fe700 3 client.4897.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:08.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.389+0000 7f04d37fe700 3 client.4897.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:08.390 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.390+0000 7f04d37fe700 20 client.4897.objecter dump_active .. 0 homeless 2023-09-30T00:53:08.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.401+0000 7f04d37fe700 1 -- 172.21.15.40:0/1915538876 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f04e40525f0 con 0x7f04ec13ac20 2023-09-30T00:53:08.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.401+0000 7f04d37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:08.401 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.401+0000 7f04d37fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:08.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.617+0000 7f04f1ab5700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:53:08.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.617+0000 7f04f1ab5700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:53:08.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.617+0000 7f04f1ab5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:08.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.617+0000 7f04f1ab5700 1 -- 172.21.15.40:0/1915538876 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04ec0a9340 con 0x7f04ec13ac20 2023-09-30T00:53:08.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.644+0000 7f04d37fe700 1 -- 172.21.15.40:0/1915538876 <== mon.1 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+11935 (secure 0 0 0) 0x7f04ec0a9340 con 0x7f04ec13ac20 2023-09-30T00:53:08.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.644+0000 7f04d37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:53:08.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.644+0000 7f04d37fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:53:08.645 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:53:08.645 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":25,"fsid":"238dba7c-9190-48bd-bb67-322ebc9ecbd0","created":"2023-09-30T00:46:45.822318+0000","modified":"2023-09-30T00:47:55.738588+0000","last_up_change":"2023-09-30T00:47:45.793188+0000","last_in_change":"2023-09-30T00:46:52.657791+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:47:40.523068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:47:51.796955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"66e1da65-310e-4463-8aa0-431896832e64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":117238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":117238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":117238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":117238}]},"public_addr":"172.21.15.40:6804/117238","cluster_addr":"172.21.15.40:6805/117238","heartbeat_back_addr":"172.21.15.40:6807/117238","heartbeat_front_addr":"172.21.15.40:6806/117238","state":["exists","up"]},{"osd":1,"uuid":"d6dd269f-2eda-4ef7-9da1-2ed5b16291b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":117237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6801","nonce":117237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":117237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":117237}]},"public_addr":"172.21.15.40:6800/117237","cluster_addr":"172.21.15.40:6801/117237","heartbeat_back_addr":"172.21.15.40:6803/117237","heartbeat_front_addr":"172.21.15.40:6802/117237","state":["exists","up"]},{"osd":2,"uuid":"8f0de27d-3674-4ff4-8451-4fcc90786c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":117239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":117239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6811","nonce":117239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":117239}]},"public_addr":"172.21.15.40:6808/117239","cluster_addr":"172.21.15.40:6809/117239","heartbeat_back_addr":"172.21.15.40:6811/117239","heartbeat_front_addr":"172.21.15.40:6810/117239","state":["exists","up"]},{"osd":3,"uuid":"d18ec2f5-7e8f-4732-a523-adc495115cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6812","nonce":117236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6813","nonce":117236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":117236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":117236}]},"public_addr":"172.21.15.40:6812/117236","cluster_addr":"172.21.15.40:6813/117236","heartbeat_back_addr":"172.21.15.40:6815/117236","heartbeat_front_addr":"172.21.15.40:6814/117236","state":["exists","up"]},{"osd":4,"uuid":"4e2465fe-bb27-47bd-8ddf-f334cac6041a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6805","nonce":117131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6806","nonce":117131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6808","nonce":117131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6807","nonce":117131}]},"public_addr":"172.21.15.135:6805/117131","cluster_addr":"172.21.15.135:6806/117131","heartbeat_back_addr":"172.21.15.135:6808/117131","heartbeat_front_addr":"172.21.15.135:6807/117131","state":["exists","up"]},{"osd":5,"uuid":"7ea449db-9b22-48b8-a61b-1937eeeb6c8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6801","nonce":117130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6802","nonce":117130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6804","nonce":117130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6803","nonce":117130}]},"public_addr":"172.21.15.135:6801/117130","cluster_addr":"172.21.15.135:6802/117130","heartbeat_back_addr":"172.21.15.135:6804/117130","heartbeat_front_addr":"172.21.15.135:6803/117130","state":["exists","up"]},{"osd":6,"uuid":"c79b4829-6393-4f9a-a9f8-d2aacb505ae9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6813","nonce":117133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6814","nonce":117133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6816","nonce":117133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6815","nonce":117133}]},"public_addr":"172.21.15.135:6813/117133","cluster_addr":"172.21.15.135:6814/117133","heartbeat_back_addr":"172.21.15.135:6816/117133","heartbeat_front_addr":"172.21.15.135:6815/117133","state":["exists","up"]},{"osd":7,"uuid":"4a205ad7-8e18-4f4d-bfcf-a30b7d9683c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6809","nonce":117132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6810","nonce":117132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6812","nonce":117132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.135:6811","nonce":117132}]},"public_addr":"172.21.15.135:6809/117132","cluster_addr":"172.21.15.135:6810/117132","heartbeat_back_addr":"172.21.15.135:6812/117132","heartbeat_front_addr":"172.21.15.135:6811/117132","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.725062+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.291589+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.713534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:41.731764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:35.977051+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:36.680016+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:35.693147+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:47:36.368009+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:53:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 10 librados: watch_flush enter 2023-09-30T00:53:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 10 librados: watch_flush exit 2023-09-30T00:53:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 20 client.4897.objecter shutdown clearing up homeless session... 2023-09-30T00:53:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 10 client.4897.objecter successfully canceled tick 2023-09-30T00:53:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 1 -- 172.21.15.40:0/1915538876 >> v2:172.21.15.135:6800/116426 conn(0x7f04d8068960 msgr2=0x7f04d806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:08.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 1 --2- 172.21.15.40:0/1915538876 >> v2:172.21.15.135:6800/116426 conn(0x7f04d8068960 0x7f04d806ae10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f04dc003a80 tx=0x7f04dc007ec0 comp rx=0 tx=0).stop 2023-09-30T00:53:08.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 10 monclient: shutdown 2023-09-30T00:53:08.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:08.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 1 -- 172.21.15.40:0/1915538876 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04ec13ac20 msgr2=0x7f04ec0a9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:08.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 1 --2- 172.21.15.40:0/1915538876 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04ec13ac20 0x7f04ec0a9620 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f04e400ecc0 tx=0x7f04e400b040 comp rx=0 tx=0).stop 2023-09-30T00:53:08.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 1 -- 172.21.15.40:0/1915538876 shutdown_connections 2023-09-30T00:53:08.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 1 --2- 172.21.15.40:0/1915538876 >> v2:172.21.15.135:6800/116426 conn(0x7f04d8068960 0x7f04d806ae10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:08.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 1 --2- 172.21.15.40:0/1915538876 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f04ec13ac20 0x7f04ec0a9620 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:08.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 1 --2- 172.21.15.40:0/1915538876 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f04ec0b1460 0x7f04ec13e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:08.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 1 --2- 172.21.15.40:0/1915538876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f04ec0b1830 0x7f04ec1516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:08.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 1 -- 172.21.15.40:0/1915538876 >> 172.21.15.40:0/1915538876 conn(0x7f04ec0631f0 msgr2=0x7f04ec0af310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:08.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 1 -- 172.21.15.40:0/1915538876 shutdown_connections 2023-09-30T00:53:08.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 1 -- 172.21.15.40:0/1915538876 wait complete. 2023-09-30T00:53:08.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:08.646+0000 7f04f1ab5700 1 librados: shutdown 2023-09-30T00:53:09.662 INFO:tasks.ceph:Scrubbing osd.0 2023-09-30T00:53:09.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:53:09.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.803+0000 7f56aafa0700 10 monclient: get_monmap_and_config 2023-09-30T00:53:09.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.803+0000 7f56aafa0700 10 monclient: build_initial_monmap 2023-09-30T00:53:09.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.803+0000 7f56aafa0700 10 monclient: monmap: 2023-09-30T00:53:09.804 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:09.804 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:09.804 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:09.804448+0000 2023-09-30T00:53:09.804 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:09.804448+0000 2023-09-30T00:53:09.804 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:09.804 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:09.804 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:09.804 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:09.804 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:09.805 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:09.805 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:09.805 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:09.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.804+0000 7f56aafa0700 1 Processor -- start 2023-09-30T00:53:09.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.804+0000 7f56aafa0700 1 -- start start 2023-09-30T00:53:09.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.804+0000 7f56aafa0700 10 monclient: init 2023-09-30T00:53:09.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.804+0000 7f56aafa0700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:09.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.804+0000 7f56aafa0700 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-09-30T00:53:09.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.804+0000 7f56aafa0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f56a40a5710 0x7f56a40a5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:09.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.804+0000 7f56aafa0700 10 monclient(hunting): picked mon.noname-a con 0x7f56a40a5710 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:09.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.804+0000 7f56aafa0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56a41415a0 addr v1:172.21.15.40:6789/0 2023-09-30T00:53:09.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.805+0000 7f56aafa0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56a413aeb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:09.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.805+0000 7f56aafa0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a413ba20 con 0x7f56a41415a0 2023-09-30T00:53:09.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.805+0000 7f56aafa0700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a40a7c40 con 0x7f56a413aeb0 2023-09-30T00:53:09.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.805+0000 7f56aafa0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:09.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.805+0000 7f56aafa0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a40a6e70 con 0x7f56a40a5710 2023-09-30T00:53:09.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.805+0000 7f56aafa0700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:09.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.805+0000 7f56aafa0700 10 monclient(hunting): authenticate will time out at 5651.106070s 2023-09-30T00:53:09.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.805+0000 7f56aa79f700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f56a413aeb0 0x7f56a413b310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:40454/0 (socket says 172.21.15.40:40454) 2023-09-30T00:53:09.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.805+0000 7f56aa79f700 1 -- 172.21.15.40:0/404769066 learned_addr learned my addr 172.21.15.40:0/404769066 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:53:09.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.806+0000 7f56a9f9e700 1 --2- 172.21.15.40:0/404769066 >> v2:172.21.15.40:3300/0 conn(0x7f56a40a5710 0x7f56a40a5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:09.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.807+0000 7f56a9f9e700 10 monclient(hunting): get_auth_request con 0x7f56a40a5710 auth_method 0 2023-09-30T00:53:09.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.807+0000 7f56a9f9e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:09.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.807+0000 7f56a9f9e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:09.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.807+0000 7f56a9f9e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:09.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.808+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646398956 0 0) 0x7f56a40a7c40 con 0x7f56a413aeb0 2023-09-30T00:53:09.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.808+0000 7f56a8f9c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:09.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.808+0000 7f56a8f9c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:09.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.808+0000 7f56a8f9c700 10 monclient(hunting): my global_id is 4900 2023-09-30T00:53:09.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.808+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5690003570 con 0x7f56a413aeb0 2023-09-30T00:53:09.810 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.810+0000 7f56a9f9e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:09.810 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.810+0000 7f56a9f9e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:09.810 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.810+0000 7f56a9f9e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:09.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.810+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118743001 0 0) 0x7f5694001110 con 0x7f56a41415a0 2023-09-30T00:53:09.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.810+0000 7f56a8f9c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:09.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.810+0000 7f56a8f9c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:09.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.810+0000 7f56a8f9c700 10 monclient(hunting): my global_id is 4809 2023-09-30T00:53:09.811 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.810+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56a40a7c40 con 0x7f56a41415a0 2023-09-30T00:53:09.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 <== mon.1 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1215882927 0 0) 0x7f5690003570 con 0x7f56a413aeb0 2023-09-30T00:53:09.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 >> v2:172.21.15.40:3300/0 conn(0x7f56a40a5710 msgr2=0x7f56a40a5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:09.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 --2- 172.21.15.40:0/404769066 >> v2:172.21.15.40:3300/0 conn(0x7f56a40a5710 0x7f56a40a5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:09.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 >> v1:172.21.15.40:6789/0 conn(0x7f56a41415a0 legacy=0x7f56a4141a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:09.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient: _finish_hunting 0 2023-09-30T00:53:09.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 monclient: found mon.noname-c-legacy 2023-09-30T00:53:09.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:09.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.135:6789/0 2023-09-30T00:53:09.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a40a6020 con 0x7f56a413aeb0 2023-09-30T00:53:09.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient: _finish_auth 0 2023-09-30T00:53:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56aafa0700 5 monclient: authenticate success, global_id 4900 2023-09-30T00:53:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 <== mon.1 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3033000540 0 0) 0x7f569c0017f0 con 0x7f56a413aeb0 2023-09-30T00:53:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56aafa0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-09-30T00:53:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient: dump: 2023-09-30T00:53:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:09.816 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:09.817 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] but i'm connected to v1:172.21.15.135:6789/0, reconnecting 2023-09-30T00:53:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 >> v1:172.21.15.135:6789/0 conn(0x7f56a413aeb0 legacy=0x7f56a413b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:53:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 --2- 172.21.15.40:0/404769066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5690003bf0 0x7f5690003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient(hunting): picked mon.a con 0x7f5690003bf0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:09.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 --2- 172.21.15.40:0/404769066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5690004810 0x7f5690004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient(hunting): picked mon.b con 0x7f5690004810 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 --2- 172.21.15.40:0/404769066 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56900051c0 0x7f569000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient(hunting): picked mon.c con 0x7f56900051c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a40a6e70 con 0x7f5690003bf0 2023-09-30T00:53:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569000c160 con 0x7f5690004810 2023-09-30T00:53:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569000c320 con 0x7f56900051c0 2023-09-30T00:53:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:09.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56a8f9c700 10 monclient(hunting): _finish_auth 0 2023-09-30T00:53:09.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56aafa0700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:09.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.813+0000 7f56aa79f700 1 --2- 172.21.15.40:0/404769066 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56900051c0 0x7f569000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:09.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.814+0000 7f56aa79f700 10 monclient(hunting): get_auth_request con 0x7f56900051c0 auth_method 0 2023-09-30T00:53:09.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.814+0000 7f56aa79f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:09.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.814+0000 7f56aa79f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:09.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.814+0000 7f56aa79f700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:53:09.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.815+0000 7f56aa79f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:09.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.815+0000 7f56aa79f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:09.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.815+0000 7f56aa79f700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:53:09.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.815+0000 7f56a979d700 1 --2- 172.21.15.40:0/404769066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5690004810 0x7f5690004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:09.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.816+0000 7f56a979d700 10 monclient(hunting): get_auth_request con 0x7f5690004810 auth_method 0 2023-09-30T00:53:09.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.816+0000 7f56a979d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:09.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.816+0000 7f56a979d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:09.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.816+0000 7f56a979d700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:53:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.816+0000 7f56a979d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.816+0000 7f56a979d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.816+0000 7f56a979d700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:53:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.818+0000 7f56aa79f700 10 monclient(hunting): handle_auth_done global_id 4900 payload 293 2023-09-30T00:53:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.818+0000 7f56aa79f700 1 -- 172.21.15.40:0/404769066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5690004810 msgr2=0x7f5690004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.818+0000 7f56aa79f700 1 --2- 172.21.15.40:0/404769066 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5690004810 0x7f5690004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.818+0000 7f56aa79f700 1 -- 172.21.15.40:0/404769066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5690003bf0 msgr2=0x7f5690003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.818+0000 7f56aa79f700 1 --2- 172.21.15.40:0/404769066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5690003bf0 0x7f5690003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.818+0000 7f56aa79f700 10 monclient: _finish_hunting 0 2023-09-30T00:53:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.818+0000 7f56aa79f700 1 monclient: found mon.c 2023-09-30T00:53:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.818+0000 7f56aa79f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:09.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.818+0000 7f56aa79f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:53:09.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.818+0000 7f56aa79f700 1 -- 172.21.15.40:0/404769066 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5690003570 con 0x7f56900051c0 2023-09-30T00:53:09.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.818+0000 7f56aa79f700 10 monclient: _finish_auth 0 2023-09-30T00:53:09.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.818+0000 7f56aa79f700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:09.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.818+0000 7f56aa79f700 1 -- 172.21.15.40:0/404769066 reap_dead start 2023-09-30T00:53:09.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.821+0000 7f56aa79f700 1 --2- 172.21.15.40:0/404769066 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56900051c0 0x7f569000ba60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f569c00a8c0 tx=0x7f569c002190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97279164f0d91cc9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:09.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.825+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c013410 con 0x7f56900051c0 2023-09-30T00:53:09.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.825+0000 7f56a8f9c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.825+0000 7f56a8f9c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:53:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.825+0000 7f56a8f9c700 10 monclient: dump: 2023-09-30T00:53:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:09.827 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:09.827 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.825+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569c013590 con 0x7f56900051c0 2023-09-30T00:53:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.825+0000 7f56aafa0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.825+0000 7f56a8f9c700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.825+0000 7f56aafa0700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.825+0000 7f56a8f9c700 1 -- 172.21.15.40:0/404769066 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c013950 con 0x7f56900051c0 2023-09-30T00:53:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.825+0000 7f56aafa0700 4 set_mon_vals no callback set 2023-09-30T00:53:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.825+0000 7f56a8f9c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.825+0000 7f56a8f9c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:53:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.825+0000 7f56a8f9c700 10 monclient: dump: 2023-09-30T00:53:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:09.828 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:09.829 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.826+0000 7f56aafa0700 10 monclient: shutdown 2023-09-30T00:53:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.826+0000 7f56aafa0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.826+0000 7f56aafa0700 1 -- 172.21.15.40:0/404769066 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56900051c0 msgr2=0x7f569000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.826+0000 7f56aafa0700 1 --2- 172.21.15.40:0/404769066 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56900051c0 0x7f569000ba60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f569c00a8c0 tx=0x7f569c002190 comp rx=0 tx=0).stop 2023-09-30T00:53:09.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.826+0000 7f56aafa0700 1 -- 172.21.15.40:0/404769066 shutdown_connections 2023-09-30T00:53:09.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.826+0000 7f56aafa0700 1 --2- 172.21.15.40:0/404769066 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56900051c0 0x7f569000ba60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:09.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.826+0000 7f56aafa0700 1 -- 172.21.15.40:0/404769066 >> 172.21.15.40:0/404769066 conn(0x7f56a406a360 msgr2=0x7f56a40631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:09.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.826+0000 7f56aafa0700 1 -- 172.21.15.40:0/404769066 shutdown_connections 2023-09-30T00:53:09.832 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.826+0000 7f56aafa0700 1 -- 172.21.15.40:0/404769066 wait complete. 2023-09-30T00:53:09.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.827+0000 7f56aafa0700 10 monclient: build_initial_monmap 2023-09-30T00:53:09.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.827+0000 7f56aafa0700 10 monclient: monmap: 2023-09-30T00:53:09.833 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:09.833 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:09.833 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:09.833 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:09.833 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:09.833 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:09.833 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:09.833 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:09.833 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:09.833 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.827+0000 7f56aafa0700 1 librados: starting msgr at 2023-09-30T00:53:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.827+0000 7f56aafa0700 1 librados: starting objecter 2023-09-30T00:53:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.827+0000 7f56aafa0700 1 Processor -- start 2023-09-30T00:53:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 1 -- start start 2023-09-30T00:53:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 1 librados: setting wanted keys 2023-09-30T00:53:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 1 librados: calling monclient init 2023-09-30T00:53:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 10 monclient: init 2023-09-30T00:53:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:53:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56900051c0 0x7f56a40ae120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 10 monclient(hunting): picked mon.noname-c con 0x7f56900051c0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56a40b53f0 0x7f56a4151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 10 monclient(hunting): picked mon.noname-b con 0x7f56a40b53f0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56a4151590 0x7f56a40b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 10 monclient(hunting): picked mon.noname-a con 0x7f56a4151590 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a413c140 con 0x7f56a4151590 2023-09-30T00:53:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a413be10 con 0x7f56a40b53f0 2023-09-30T00:53:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a40b4c40 con 0x7f56900051c0 2023-09-30T00:53:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.828+0000 7f56aafa0700 10 monclient(hunting): authenticate will time out at 5651.129592s 2023-09-30T00:53:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.829+0000 7f56aa79f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56a4151590 0x7f56a40b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.829+0000 7f56a979d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56a40b53f0 0x7f56a4151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.829+0000 7f56aa79f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56a4151590 0x7f56a40b26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49996/0 (socket says 172.21.15.40:49996) 2023-09-30T00:53:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.829+0000 7f56aa79f700 1 -- 172.21.15.40:0/4286121249 learned_addr learned my addr 172.21.15.40:0/4286121249 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.829+0000 7f56aa79f700 10 monclient(hunting): get_auth_request con 0x7f56a4151590 auth_method 0 2023-09-30T00:53:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.829+0000 7f56aa79f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.829+0000 7f56aa79f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.829+0000 7f56aa79f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.829+0000 7f56a979d700 10 monclient(hunting): get_auth_request con 0x7f56a40b53f0 auth_method 0 2023-09-30T00:53:09.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.829+0000 7f56a979d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:09.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.829+0000 7f56a979d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:09.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.829+0000 7f56a979d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:09.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.830+0000 7f56a979d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:09.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.830+0000 7f56a979d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:09.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.830+0000 7f56a979d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:09.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.830+0000 7f56aa79f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:09.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.830+0000 7f56aa79f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:09.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.830+0000 7f56aa79f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:09.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.830+0000 7f56a9f9e700 1 --2- 172.21.15.40:0/4286121249 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56900051c0 0x7f56a40ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:09.837 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.830+0000 7f56a9f9e700 10 monclient(hunting): get_auth_request con 0x7f56900051c0 auth_method 0 2023-09-30T00:53:09.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.830+0000 7f56a9f9e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:09.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.830+0000 7f56a9f9e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:09.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.830+0000 7f56a9f9e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:09.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.831+0000 7f56a9f9e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:09.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.831+0000 7f56a9f9e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:09.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.831+0000 7f56a9f9e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:09.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.833+0000 7f56a979d700 10 monclient(hunting): handle_auth_done global_id 4903 payload 824 2023-09-30T00:53:09.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.833+0000 7f56a979d700 1 -- 172.21.15.40:0/4286121249 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56900051c0 msgr2=0x7f56a40ae120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:09.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.833+0000 7f56a979d700 1 --2- 172.21.15.40:0/4286121249 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f56900051c0 0x7f56a40ae120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:09.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.833+0000 7f56a979d700 1 -- 172.21.15.40:0/4286121249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56a4151590 msgr2=0x7f56a40b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:09.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.833+0000 7f56a979d700 1 --2- 172.21.15.40:0/4286121249 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56a4151590 0x7f56a40b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:09.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.833+0000 7f56a979d700 10 monclient: _finish_hunting 0 2023-09-30T00:53:09.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.833+0000 7f56a979d700 1 monclient: found mon.noname-b 2023-09-30T00:53:09.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.833+0000 7f56a979d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:09.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.833+0000 7f56a979d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:09.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.833+0000 7f56a979d700 1 -- 172.21.15.40:0/4286121249 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a40b3a10 con 0x7f56a40b53f0 2023-09-30T00:53:09.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.835+0000 7f56a979d700 1 --2- 172.21.15.40:0/4286121249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56a40b53f0 0x7f56a4151050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5694000c00 tx=0x7f5694007d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:09.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.835+0000 7f5699ffb700 10 client.?.objecter ms_handle_connect 0x7f56a40b53f0 2023-09-30T00:53:09.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.835+0000 7f5699ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:09.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f5699ffb700 1 -- 172.21.15.40:0/4286121249 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5694001110 con 0x7f56a40b53f0 2023-09-30T00:53:09.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f5699ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:09.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f5699ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:09.841 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f5699ffb700 10 monclient: dump: 2023-09-30T00:53:09.841 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:09.841 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:09.842 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:09.842 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:09.842 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:09.842 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:09.842 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:09.842 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:09.842 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:09.842 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:09.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f5699ffb700 10 monclient: _finish_auth 0 2023-09-30T00:53:09.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f5699ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:09.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f5699ffb700 1 -- 172.21.15.40:0/4286121249 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5694013600 con 0x7f56a40b53f0 2023-09-30T00:53:09.842 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f56aafa0700 5 monclient: authenticate success, global_id 4903 2023-09-30T00:53:09.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f5699ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:09.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f5699ffb700 1 -- 172.21.15.40:0/4286121249 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56940139c0 con 0x7f56a40b53f0 2023-09-30T00:53:09.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f5699ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:09.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f5699ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:09.843 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f5699ffb700 10 monclient: dump: 2023-09-30T00:53:09.843 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:09.843 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:09.843 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:09.843 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:09.843 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:09.843 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:09.843 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:09.844 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f56aafa0700 10 monclient: _renew_subs 2023-09-30T00:53:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f56aafa0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f56aafa0700 1 -- 172.21.15.40:0/4286121249 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a413ec90 con 0x7f56a40b53f0 2023-09-30T00:53:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f569bfff700 4 set_mon_vals no callback set 2023-09-30T00:53:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f56aafa0700 10 client.4903.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f56aafa0700 10 monclient: _renew_subs 2023-09-30T00:53:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f56aafa0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f56aafa0700 1 -- 172.21.15.40:0/4286121249 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a40b2c30 con 0x7f56a40b53f0 2023-09-30T00:53:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.839+0000 7f56aafa0700 1 librados: init done 2023-09-30T00:53:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.840+0000 7f56aafa0700 10 client.4903.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.840+0000 7f56aafa0700 15 client.4903.objecter _session_command_op_assign -1 1 2023-09-30T00:53:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.840+0000 7f56aafa0700 20 client.4903.objecter _get_session osd=-1 returning homeless 2023-09-30T00:53:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.840+0000 7f56aafa0700 10 client.4903.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.840+0000 7f56aafa0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.840+0000 7f56aafa0700 1 -- 172.21.15.40:0/4286121249 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f56a4064a00 con 0x7f56a40b53f0 2023-09-30T00:53:09.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.859+0000 7f5699ffb700 1 -- 172.21.15.40:0/4286121249 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5694013d30 con 0x7f56a40b53f0 2023-09-30T00:53:09.859 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.859+0000 7f5699ffb700 1 --2- 172.21.15.40:0/4286121249 >> v2:172.21.15.135:6800/116426 conn(0x7f5688068960 0x7f568806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:09.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.859+0000 7f56a9f9e700 1 --2- 172.21.15.40:0/4286121249 >> v2:172.21.15.135:6800/116426 conn(0x7f5688068960 0x7f568806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:09.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.859+0000 7f56a9f9e700 10 monclient: get_auth_request con 0x7f5688068960 auth_method 0 2023-09-30T00:53:09.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.860+0000 7f56a9f9e700 1 --2- 172.21.15.40:0/4286121249 >> v2:172.21.15.135:6800/116426 conn(0x7f5688068960 0x7f568806ae10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f56a0009170 tx=0x7f56a00091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:09.860 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.860+0000 7f5699ffb700 10 client.4903.objecter ms_handle_connect 0x7f5688068960 2023-09-30T00:53:09.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 1 -- 172.21.15.40:0/4286121249 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f56940560e0 con 0x7f56a40b53f0 2023-09-30T00:53:09.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 10 client.4903.objecter ms_dispatch 0x7f56a4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:09.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 3 client.4903.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:09.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 3 client.4903.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:09.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 10 client.4903.objecter checking command 1 2023-09-30T00:53:09.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 1 --2- 172.21.15.40:0/4286121249 >> v2:172.21.15.40:6804/117238 conn(0x7f568806e8d0 0x7f5688070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:09.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 20 client.4903.objecter _get_session s=0x7f568806e220 osd=0 3 2023-09-30T00:53:09.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 20 client.4903.objecter put_session s=0x7f568806e220 osd=0 3 2023-09-30T00:53:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 15 client.4903.objecter _session_command_op_remove -1 1 2023-09-30T00:53:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 20 client.4903.objecter _get_session s=0x7f568806e220 osd=0 3 2023-09-30T00:53:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 20 client.4903.objecter get_session s=0x7f568806e220 osd=0 3 2023-09-30T00:53:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 15 client.4903.objecter _session_command_op_assign 0 1 2023-09-30T00:53:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 20 client.4903.objecter put_session s=0x7f568806e220 osd=0 4 2023-09-30T00:53:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 10 client.4903.objecter _send_command 1 2023-09-30T00:53:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 1 -- 172.21.15.40:0/4286121249 --> v2:172.21.15.40:6804/117238 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5688071350 con 0x7f568806e8d0 2023-09-30T00:53:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.862+0000 7f5699ffb700 20 client.4903.objecter dump_active .. 0 homeless 2023-09-30T00:53:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.863+0000 7f56aa79f700 1 --2- 172.21.15.40:0/4286121249 >> v2:172.21.15.40:6804/117238 conn(0x7f568806e8d0 0x7f5688070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:09.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.863+0000 7f56aa79f700 10 monclient: get_auth_request con 0x7f568806e8d0 auth_method 0 2023-09-30T00:53:09.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.864+0000 7f5699ffb700 1 -- 172.21.15.40:0/4286121249 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5694051d20 con 0x7f56a40b53f0 2023-09-30T00:53:09.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.864+0000 7f5699ffb700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:53:09.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.867+0000 7f5699ffb700 10 client.4903.objecter ms_handle_connect 0x7f568806e8d0 2023-09-30T00:53:09.867 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.867+0000 7f56aa79f700 1 --2- 172.21.15.40:0/4286121249 >> v2:172.21.15.40:6804/117238 conn(0x7f568806e8d0 0x7f5688070ce0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:09.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.894+0000 7f5699ffb700 1 -- 172.21.15.40:0/4286121249 <== osd.0 v2:172.21.15.40:6804/117238 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f5688071350 con 0x7f568806e8d0 2023-09-30T00:53:09.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.894+0000 7f5699ffb700 10 client.4903.objecter ms_dispatch 0x7f56a4001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:53:09.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.894+0000 7f5699ffb700 10 client.4903.objecter _finish_command 1 = system:0 2023-09-30T00:53:09.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.894+0000 7f5699ffb700 20 client.4903.objecter put_session s=0x7f568806e220 osd=0 4 2023-09-30T00:53:09.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.894+0000 7f5699ffb700 15 client.4903.objecter _session_command_op_remove 0 1 2023-09-30T00:53:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.911+0000 7f56aafa0700 10 client.4903.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T00:53:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.911+0000 7f56aafa0700 15 client.4903.objecter _session_command_op_assign -1 2 2023-09-30T00:53:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.911+0000 7f56aafa0700 20 client.4903.objecter _get_session s=0x7f568806e220 osd=0 3 2023-09-30T00:53:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.911+0000 7f56aafa0700 20 client.4903.objecter put_session s=0x7f568806e220 osd=0 3 2023-09-30T00:53:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.911+0000 7f56aafa0700 20 client.4903.objecter _get_session s=0x7f568806e220 osd=0 3 2023-09-30T00:53:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.911+0000 7f56aafa0700 15 client.4903.objecter _session_command_op_remove -1 2 2023-09-30T00:53:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.911+0000 7f56aafa0700 20 client.4903.objecter get_session s=0x7f568806e220 osd=0 3 2023-09-30T00:53:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.911+0000 7f56aafa0700 15 client.4903.objecter _session_command_op_assign 0 2 2023-09-30T00:53:09.912 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.911+0000 7f56aafa0700 20 client.4903.objecter put_session s=0x7f568806e220 osd=0 4 2023-09-30T00:53:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.911+0000 7f56aafa0700 10 client.4903.objecter _send_command 2 2023-09-30T00:53:09.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.911+0000 7f56aafa0700 1 -- 172.21.15.40:0/4286121249 --> v2:172.21.15.40:6804/117238 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f56a413bfa0 con 0x7f568806e8d0 2023-09-30T00:53:09.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.937+0000 7f5699ffb700 1 -- 172.21.15.40:0/4286121249 <== osd.0 v2:172.21.15.40:6804/117238 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f569c019040 con 0x7f568806e8d0 2023-09-30T00:53:09.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.937+0000 7f5699ffb700 10 client.4903.objecter ms_dispatch 0x7f56a4001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:53:09.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.937+0000 7f5699ffb700 10 client.4903.objecter _finish_command 2 = system:0 2023-09-30T00:53:09.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.937+0000 7f5699ffb700 20 client.4903.objecter put_session s=0x7f568806e220 osd=0 4 2023-09-30T00:53:09.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.937+0000 7f5699ffb700 15 client.4903.objecter _session_command_op_remove 0 2 2023-09-30T00:53:09.938 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-09-30T00:53:09.939 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:53:09.939 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-09-30T00:53:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 10 librados: watch_flush enter 2023-09-30T00:53:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 10 librados: watch_flush exit 2023-09-30T00:53:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 10 client.4903.objecter close_session for osd.0 2023-09-30T00:53:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 1 -- 172.21.15.40:0/4286121249 >> v2:172.21.15.40:6804/117238 conn(0x7f568806e8d0 msgr2=0x7f5688070ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 1 --2- 172.21.15.40:0/4286121249 >> v2:172.21.15.40:6804/117238 conn(0x7f568806e8d0 0x7f5688070ce0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 20 client.4903.objecter put_session s=0x7f568806e220 osd=0 1 2023-09-30T00:53:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 20 client.4903.objecter shutdown clearing up homeless session... 2023-09-30T00:53:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 10 client.4903.objecter successfully canceled tick 2023-09-30T00:53:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 1 -- 172.21.15.40:0/4286121249 >> v2:172.21.15.135:6800/116426 conn(0x7f5688068960 msgr2=0x7f568806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:09.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 1 --2- 172.21.15.40:0/4286121249 >> v2:172.21.15.135:6800/116426 conn(0x7f5688068960 0x7f568806ae10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f56a0009170 tx=0x7f56a00091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 10 monclient: shutdown 2023-09-30T00:53:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 1 -- 172.21.15.40:0/4286121249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56a40b53f0 msgr2=0x7f56a4151050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 1 --2- 172.21.15.40:0/4286121249 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f56a40b53f0 0x7f56a4151050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5694000c00 tx=0x7f5694007d30 comp rx=0 tx=0).stop 2023-09-30T00:53:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aa79f700 1 -- 172.21.15.40:0/4286121249 reap_dead start 2023-09-30T00:53:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 1 -- 172.21.15.40:0/4286121249 shutdown_connections 2023-09-30T00:53:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 1 -- 172.21.15.40:0/4286121249 >> 172.21.15.40:0/4286121249 conn(0x7f56a406a360 msgr2=0x7f56a4070fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 1 -- 172.21.15.40:0/4286121249 shutdown_connections 2023-09-30T00:53:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 1 -- 172.21.15.40:0/4286121249 wait complete. 2023-09-30T00:53:09.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:09.938+0000 7f56aafa0700 1 librados: shutdown 2023-09-30T00:53:09.953 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-30T00:53:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.084+0000 7f61b89a0700 10 monclient: get_monmap_and_config 2023-09-30T00:53:10.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.084+0000 7f61b89a0700 10 monclient: build_initial_monmap 2023-09-30T00:53:10.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.084+0000 7f61b89a0700 10 monclient: monmap: 2023-09-30T00:53:10.085 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:10.085 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:10.085 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:10.085456+0000 2023-09-30T00:53:10.085 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:10.085456+0000 2023-09-30T00:53:10.085 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:10.085 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:10.085 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:10.085 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:10.086 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:10.086 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:10.086 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:10.086 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:10.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 1 Processor -- start 2023-09-30T00:53:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 1 -- start start 2023-09-30T00:53:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 10 monclient: init 2023-09-30T00:53:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-09-30T00:53:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f61b40b1460 0x7f61b40b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 10 monclient(hunting): picked mon.noname-c con 0x7f61b40b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61b40a5fb0 0x7f61b4141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 10 monclient(hunting): picked mon.noname-b con 0x7f61b40a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61b40af410 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b40a71f0 con 0x7f61b40af410 2023-09-30T00:53:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b40a74d0 con 0x7f61b40b1460 2023-09-30T00:53:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.085+0000 7f61b89a0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40a63c0 con 0x7f61b40a5fb0 2023-09-30T00:53:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.086+0000 7f61b89a0700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.086+0000 7f61b89a0700 10 monclient(hunting): authenticate will time out at 5651.387012s 2023-09-30T00:53:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.086+0000 7f61b1d9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61b40a5fb0 0x7f61b4141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.086+0000 7f61b1d9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61b40a5fb0 0x7f61b4141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:54950/0 (socket says 172.21.15.40:54950) 2023-09-30T00:53:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.087+0000 7f61b1d9b700 1 -- 172.21.15.40:0/1132081747 learned_addr learned my addr 172.21.15.40:0/1132081747 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.087+0000 7f61b1d9b700 10 monclient(hunting): get_auth_request con 0x7f61b40a5fb0 auth_method 0 2023-09-30T00:53:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.087+0000 7f61b1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.087+0000 7f61b1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:10.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.087+0000 7f61b1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:10.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.087+0000 7f61b259c700 1 --2- 172.21.15.40:0/1132081747 >> v2:172.21.15.135:3300/0 conn(0x7f61b40b1460 0x7f61b40b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:10.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.087+0000 7f61b1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:10.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.087+0000 7f61b1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:10.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.087+0000 7f61b1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:10.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.087+0000 7f61b259c700 10 monclient(hunting): get_auth_request con 0x7f61b40b1460 auth_method 0 2023-09-30T00:53:10.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.087+0000 7f61b259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:10.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.087+0000 7f61b259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:10.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.087+0000 7f61b259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:10.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.088+0000 7f61b259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:10.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.088+0000 7f61b259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:10.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.088+0000 7f61b259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:10.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.089+0000 7f61b1d9b700 10 monclient(hunting): handle_auth_done global_id 4703 payload 274 2023-09-30T00:53:10.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.089+0000 7f61b1d9b700 1 -- 172.21.15.40:0/1132081747 >> v2:172.21.15.135:3300/0 conn(0x7f61b40b1460 msgr2=0x7f61b40b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:10.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.089+0000 7f61b1d9b700 1 --2- 172.21.15.40:0/1132081747 >> v2:172.21.15.135:3300/0 conn(0x7f61b40b1460 0x7f61b40b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.089+0000 7f61b1d9b700 1 -- 172.21.15.40:0/1132081747 >> v1:172.21.15.135:6789/0 conn(0x7f61b40af410 legacy=0x7f61b40af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:10.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.089+0000 7f61b1d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:53:10.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.089+0000 7f61b1d9b700 1 monclient: found mon.noname-b 2023-09-30T00:53:10.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.089+0000 7f61b1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:10.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.089+0000 7f61b1d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:53:10.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.089+0000 7f61b1d9b700 1 -- 172.21.15.40:0/1132081747 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b40a56d0 con 0x7f61b40a5fb0 2023-09-30T00:53:10.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.091+0000 7f61b1d9b700 1 --2- 172.21.15.40:0/1132081747 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61b40a5fb0 0x7f61b4141730 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f61a80099b0 tx=0x7f61a8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=746f5c86a48b520e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:10.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.094+0000 7f61b159a700 1 -- 172.21.15.40:0/1132081747 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a8009ea0 con 0x7f61b40a5fb0 2023-09-30T00:53:10.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.094+0000 7f61b159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:10.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.094+0000 7f61b159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:10.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.094+0000 7f61b159a700 10 monclient: dump: 2023-09-30T00:53:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:10.095 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.094+0000 7f61b159a700 10 monclient: _finish_auth 0 2023-09-30T00:53:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.094+0000 7f61b159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:10.095 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.094+0000 7f61b89a0700 5 monclient: authenticate success, global_id 4703 2023-09-30T00:53:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.094+0000 7f61b89a0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b159a700 1 -- 172.21.15.40:0/1132081747 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a800b860 con 0x7f61b40a5fb0 2023-09-30T00:53:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b159a700 1 -- 172.21.15.40:0/1132081747 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a800bc00 con 0x7f61b40a5fb0 2023-09-30T00:53:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:53:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b159a700 10 monclient: dump: 2023-09-30T00:53:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:10.097 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:10.098 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:10.098 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:10.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b89a0700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:10.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b89a0700 4 set_mon_vals no callback set 2023-09-30T00:53:10.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b89a0700 10 monclient: shutdown 2023-09-30T00:53:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b89a0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b89a0700 1 -- 172.21.15.40:0/1132081747 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61b40a5fb0 msgr2=0x7f61b4141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b89a0700 1 --2- 172.21.15.40:0/1132081747 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61b40a5fb0 0x7f61b4141730 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f61a80099b0 tx=0x7f61a8009250 comp rx=0 tx=0).stop 2023-09-30T00:53:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b89a0700 1 -- 172.21.15.40:0/1132081747 shutdown_connections 2023-09-30T00:53:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b89a0700 1 --2- 172.21.15.40:0/1132081747 >> v2:172.21.15.135:3300/0 conn(0x7f61b40b1460 0x7f61b40b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b89a0700 1 --2- 172.21.15.40:0/1132081747 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61b40a5fb0 0x7f61b4141730 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.096+0000 7f61b89a0700 1 -- 172.21.15.40:0/1132081747 >> 172.21.15.40:0/1132081747 conn(0x7f61b40631f0 msgr2=0x7f61b40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.097+0000 7f61b89a0700 1 -- 172.21.15.40:0/1132081747 shutdown_connections 2023-09-30T00:53:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.097+0000 7f61b89a0700 1 -- 172.21.15.40:0/1132081747 wait complete. 2023-09-30T00:53:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.097+0000 7f61b89a0700 10 monclient: build_initial_monmap 2023-09-30T00:53:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.097+0000 7f61b89a0700 10 monclient: monmap: 2023-09-30T00:53:10.101 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:10.102 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.098+0000 7f61b89a0700 1 librados: starting msgr at 2023-09-30T00:53:10.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.098+0000 7f61b89a0700 1 librados: starting objecter 2023-09-30T00:53:10.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.098+0000 7f61b89a0700 1 Processor -- start 2023-09-30T00:53:10.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.098+0000 7f61b89a0700 1 -- start start 2023-09-30T00:53:10.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.098+0000 7f61b89a0700 1 librados: setting wanted keys 2023-09-30T00:53:10.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.098+0000 7f61b89a0700 1 librados: calling monclient init 2023-09-30T00:53:10.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.098+0000 7f61b89a0700 10 monclient: init 2023-09-30T00:53:10.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:10.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:53:10.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61b40a5fb0 0x7f61b40a8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:10.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 10 monclient(hunting): picked mon.noname-c con 0x7f61b40a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:10.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61b40af410 0x7f61b40a8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:10.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 10 monclient(hunting): picked mon.noname-a con 0x7f61b40af410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:10.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61b40b1460 0x7f61b413d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:10.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 10 monclient(hunting): picked mon.noname-b con 0x7f61b40b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:10.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:10.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40a9070 con 0x7f61b40af410 2023-09-30T00:53:10.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:10.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40a9310 con 0x7f61b40b1460 2023-09-30T00:53:10.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:10.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b413f060 con 0x7f61b40a5fb0 2023-09-30T00:53:10.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:10.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b89a0700 10 monclient(hunting): authenticate will time out at 5651.400423s 2023-09-30T00:53:10.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.099+0000 7f61b259c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61b40a5fb0 0x7f61b40a8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:10.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b2d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61b40b1460 0x7f61b413d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:10.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b1d9b700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61b40af410 0x7f61b40a8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:10.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b259c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61b40a5fb0 0x7f61b40a8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:54952/0 (socket says 172.21.15.40:54952) 2023-09-30T00:53:10.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b259c700 1 -- 172.21.15.40:0/4088409089 learned_addr learned my addr 172.21.15.40:0/4088409089 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:10.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b259c700 10 monclient(hunting): get_auth_request con 0x7f61b40a5fb0 auth_method 0 2023-09-30T00:53:10.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:10.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:10.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:10.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b2d9d700 10 monclient(hunting): get_auth_request con 0x7f61b40b1460 auth_method 0 2023-09-30T00:53:10.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:10.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:10.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:10.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b1d9b700 10 monclient(hunting): get_auth_request con 0x7f61b40af410 auth_method 0 2023-09-30T00:53:10.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:10.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:10.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:10.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:10.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.100+0000 7f61b2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:10.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.101+0000 7f61b1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:10.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.101+0000 7f61b1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:10.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.101+0000 7f61b1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:10.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.103+0000 7f61b2d9d700 10 monclient(hunting): handle_auth_done global_id 4912 payload 824 2023-09-30T00:53:10.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.103+0000 7f61b2d9d700 1 -- 172.21.15.40:0/4088409089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61b40a5fb0 msgr2=0x7f61b40a8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:10.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.103+0000 7f61b2d9d700 1 --2- 172.21.15.40:0/4088409089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61b40a5fb0 0x7f61b40a8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.103+0000 7f61b2d9d700 1 -- 172.21.15.40:0/4088409089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61b40af410 msgr2=0x7f61b40a8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:10.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.103+0000 7f61b2d9d700 1 --2- 172.21.15.40:0/4088409089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61b40af410 0x7f61b40a8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.104+0000 7f61b2d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:53:10.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.104+0000 7f61b2d9d700 1 monclient: found mon.noname-b 2023-09-30T00:53:10.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.104+0000 7f61b2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:10.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.104+0000 7f61b2d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:10.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.104+0000 7f61b2d9d700 1 -- 172.21.15.40:0/4088409089 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a80094f0 con 0x7f61b40b1460 2023-09-30T00:53:10.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.106+0000 7f61b2d9d700 1 --2- 172.21.15.40:0/4088409089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61b40b1460 0x7f61b413d150 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00c830 tx=0x7f61ac001690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:10.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.106+0000 7f61a27fc700 10 client.?.objecter ms_handle_connect 0x7f61b40b1460 2023-09-30T00:53:10.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.106+0000 7f61a27fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:10.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61a27fc700 1 -- 172.21.15.40:0/4088409089 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac012630 con 0x7f61b40b1460 2023-09-30T00:53:10.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:10.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61a27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:10.111 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61a27fc700 10 monclient: dump: 2023-09-30T00:53:10.111 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:10.112 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61a27fc700 10 monclient: _finish_auth 0 2023-09-30T00:53:10.112 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61a27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61a27fc700 1 -- 172.21.15.40:0/4088409089 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac012790 con 0x7f61b40b1460 2023-09-30T00:53:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61a27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61a27fc700 1 -- 172.21.15.40:0/4088409089 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac012b50 con 0x7f61b40b1460 2023-09-30T00:53:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61a27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61a27fc700 10 monclient: dump: 2023-09-30T00:53:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:10.113 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:10.114 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61b89a0700 5 monclient: authenticate success, global_id 4912 2023-09-30T00:53:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61b89a0700 10 monclient: _renew_subs 2023-09-30T00:53:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61b89a0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61b89a0700 1 -- 172.21.15.40:0/4088409089 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b40ae4e0 con 0x7f61b40b1460 2023-09-30T00:53:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61b0d99700 4 set_mon_vals no callback set 2023-09-30T00:53:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61b89a0700 10 client.4912.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:10.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61b89a0700 10 monclient: _renew_subs 2023-09-30T00:53:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61b89a0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61b89a0700 1 -- 172.21.15.40:0/4088409089 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b413d850 con 0x7f61b40b1460 2023-09-30T00:53:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.110+0000 7f61b89a0700 1 librados: init done 2023-09-30T00:53:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.111+0000 7f61b89a0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.111+0000 7f61b89a0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.111+0000 7f61b89a0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:10.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.111+0000 7f61b89a0700 1 -- 172.21.15.40:0/4088409089 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b40a9490 con 0x7f61b40b1460 2023-09-30T00:53:10.132 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.132+0000 7f61a27fc700 1 -- 172.21.15.40:0/4088409089 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f61ac024420 con 0x7f61b40b1460 2023-09-30T00:53:10.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.132+0000 7f61a27fc700 1 --2- 172.21.15.40:0/4088409089 >> v2:172.21.15.135:6800/116426 conn(0x7f61940688c0 0x7f619406ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:10.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.133+0000 7f61b259c700 1 --2- 172.21.15.40:0/4088409089 >> v2:172.21.15.135:6800/116426 conn(0x7f61940688c0 0x7f619406ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:10.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.133+0000 7f61b259c700 10 monclient: get_auth_request con 0x7f61940688c0 auth_method 0 2023-09-30T00:53:10.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.133+0000 7f61b259c700 1 --2- 172.21.15.40:0/4088409089 >> v2:172.21.15.135:6800/116426 conn(0x7f61940688c0 0x7f619406ad70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009dd0 tx=0x7f61a40091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:10.133 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.133+0000 7f61a27fc700 10 client.4912.objecter ms_handle_connect 0x7f61940688c0 2023-09-30T00:53:10.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.138+0000 7f61a27fc700 1 -- 172.21.15.40:0/4088409089 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f61ac056260 con 0x7f61b40b1460 2023-09-30T00:53:10.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.138+0000 7f61a27fc700 10 client.4912.objecter ms_dispatch 0x7f61b4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:10.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.138+0000 7f61a27fc700 3 client.4912.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:10.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.138+0000 7f61a27fc700 3 client.4912.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:10.138 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.138+0000 7f61a27fc700 20 client.4912.objecter dump_active .. 0 homeless 2023-09-30T00:53:10.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.382+0000 7f61a27fc700 1 -- 172.21.15.40:0/4088409089 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61ac051cd0 con 0x7f61b40b1460 2023-09-30T00:53:10.382 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.382+0000 7f61a27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:10.383 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.382+0000 7f61a27fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:10.615 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.615+0000 7f61b89a0700 1 -- 172.21.15.40:0/4088409089 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f61b40a9490 con 0x7f61940688c0 2023-09-30T00:53:10.616 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.615+0000 7f61a27fc700 1 -- 172.21.15.40:0/4088409089 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f61b40a9490 con 0x7f61940688c0 2023-09-30T00:53:10.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 10 librados: watch_flush enter 2023-09-30T00:53:10.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 10 librados: watch_flush exit 2023-09-30T00:53:10.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 20 client.4912.objecter shutdown clearing up homeless session... 2023-09-30T00:53:10.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 10 client.4912.objecter successfully canceled tick 2023-09-30T00:53:10.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 1 -- 172.21.15.40:0/4088409089 >> v2:172.21.15.135:6800/116426 conn(0x7f61940688c0 msgr2=0x7f619406ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:10.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 1 --2- 172.21.15.40:0/4088409089 >> v2:172.21.15.135:6800/116426 conn(0x7f61940688c0 0x7f619406ad70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009dd0 tx=0x7f61a40091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 10 monclient: shutdown 2023-09-30T00:53:10.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:10.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 1 -- 172.21.15.40:0/4088409089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61b40b1460 msgr2=0x7f61b413d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:10.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 1 --2- 172.21.15.40:0/4088409089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61b40b1460 0x7f61b413d150 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f61ac00c830 tx=0x7f61ac001690 comp rx=0 tx=0).stop 2023-09-30T00:53:10.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 1 -- 172.21.15.40:0/4088409089 shutdown_connections 2023-09-30T00:53:10.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 1 --2- 172.21.15.40:0/4088409089 >> v2:172.21.15.135:6800/116426 conn(0x7f61940688c0 0x7f619406ad70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 1 --2- 172.21.15.40:0/4088409089 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f61b40b1460 0x7f61b413d150 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 1 --2- 172.21.15.40:0/4088409089 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f61b40a5fb0 0x7f61b40a8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 1 --2- 172.21.15.40:0/4088409089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f61b40af410 0x7f61b40a8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 1 -- 172.21.15.40:0/4088409089 >> 172.21.15.40:0/4088409089 conn(0x7f61b40631f0 msgr2=0x7f61b4138cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:10.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 1 -- 172.21.15.40:0/4088409089 shutdown_connections 2023-09-30T00:53:10.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 1 -- 172.21.15.40:0/4088409089 wait complete. 2023-09-30T00:53:10.620 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.618+0000 7f61b89a0700 1 librados: shutdown 2023-09-30T00:53:10.620 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 0 to deep-scrub 2023-09-30T00:53:10.632 INFO:tasks.ceph:Scrubbing osd.1 2023-09-30T00:53:10.632 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:53:10.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.771+0000 7f9ae86f0700 10 monclient: get_monmap_and_config 2023-09-30T00:53:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.771+0000 7f9ae86f0700 10 monclient: build_initial_monmap 2023-09-30T00:53:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.771+0000 7f9ae86f0700 10 monclient: monmap: 2023-09-30T00:53:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:10.772534+0000 2023-09-30T00:53:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:10.772534+0000 2023-09-30T00:53:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:10.773 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:10.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.772+0000 7f9ae86f0700 1 Processor -- start 2023-09-30T00:53:10.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.772+0000 7f9ae86f0700 1 -- start start 2023-09-30T00:53:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.772+0000 7f9ae86f0700 10 monclient: init 2023-09-30T00:53:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.772+0000 7f9ae86f0700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.772+0000 7f9ae86f0700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2023-09-30T00:53:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.772+0000 7f9ae86f0700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9ae006bd70 0x7f9ae006e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.772+0000 7f9ae86f0700 10 monclient(hunting): picked mon.noname-a con 0x7f9ae006bd70 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.773+0000 7f9ae86f0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ae006c140 addr v1:172.21.15.40:6789/0 2023-09-30T00:53:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.773+0000 7f9ae86f0700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9ae013e2c0 0x7f9ae0144f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.773+0000 7f9ae86f0700 10 monclient(hunting): picked mon.noname-c con 0x7f9ae013e2c0 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.773+0000 7f9ae86f0700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae006d320 con 0x7f9ae006c140 2023-09-30T00:53:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.773+0000 7f9ae86f0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.773+0000 7f9ae86f0700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae006d590 con 0x7f9ae006bd70 2023-09-30T00:53:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.773+0000 7f9ae86f0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.773+0000 7f9ae86f0700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae006d870 con 0x7f9ae013e2c0 2023-09-30T00:53:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.773+0000 7f9ae86f0700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.773+0000 7f9ae86f0700 10 monclient(hunting): authenticate will time out at 5652.074085s 2023-09-30T00:53:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.773+0000 7f9ae6c8d700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9ae013e2c0 0x7f9ae0144f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.774+0000 7f9ae6c8d700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9ae013e2c0 0x7f9ae0144f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:33686/0 (socket says 172.21.15.40:33686) 2023-09-30T00:53:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.774+0000 7f9ae6c8d700 1 -- 172.21.15.40:0/3988675406 learned_addr learned my addr 172.21.15.40:0/3988675406 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.774+0000 7f9ae6c8d700 10 monclient(hunting): get_auth_request con 0x7f9ae013e2c0 auth_method 0 2023-09-30T00:53:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.774+0000 7f9ae6c8d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.774+0000 7f9ae6c8d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.774+0000 7f9ae6c8d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.774+0000 7f9ae648c700 1 --2- 172.21.15.40:0/3988675406 >> v2:172.21.15.40:3300/0 conn(0x7f9ae006bd70 0x7f9ae006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.774+0000 7f9ae6c8d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:10.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.774+0000 7f9ae6c8d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:10.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.774+0000 7f9ae6c8d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:10.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.774+0000 7f9ae648c700 10 monclient(hunting): get_auth_request con 0x7f9ae006bd70 auth_method 0 2023-09-30T00:53:10.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.774+0000 7f9ae648c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:10.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.774+0000 7f9ae648c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:10.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.774+0000 7f9ae648c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:10.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.776+0000 7f9ae6c8d700 10 monclient(hunting): handle_auth_done global_id 4915 payload 274 2023-09-30T00:53:10.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.776+0000 7f9ae6c8d700 1 -- 172.21.15.40:0/3988675406 >> v2:172.21.15.40:3300/0 conn(0x7f9ae006bd70 msgr2=0x7f9ae006e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:10.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.776+0000 7f9ae6c8d700 1 --2- 172.21.15.40:0/3988675406 >> v2:172.21.15.40:3300/0 conn(0x7f9ae006bd70 0x7f9ae006e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.776+0000 7f9ae6c8d700 1 -- 172.21.15.40:0/3988675406 >> v1:172.21.15.40:6789/0 conn(0x7f9ae006c140 legacy=0x7f9ae013dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:10.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.776+0000 7f9ae6c8d700 10 monclient: _finish_hunting 0 2023-09-30T00:53:10.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.776+0000 7f9ae6c8d700 1 monclient: found mon.noname-c 2023-09-30T00:53:10.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.776+0000 7f9ae6c8d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:10.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.776+0000 7f9ae6c8d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:53:10.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.776+0000 7f9ae6c8d700 1 -- 172.21.15.40:0/3988675406 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae006de30 con 0x7f9ae013e2c0 2023-09-30T00:53:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.778+0000 7f9ae6c8d700 1 --2- 172.21.15.40:0/3988675406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ae013e2c0 0x7f9ae0144f50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9ae013b690 tx=0x7f9ad800ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2ed27d113f03a23 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:10.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae548a700 1 -- 172.21.15.40:0/3988675406 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8010070 con 0x7f9ae013e2c0 2023-09-30T00:53:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae548a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae548a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:53:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae548a700 10 monclient: dump: 2023-09-30T00:53:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:10.784 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae548a700 10 monclient: _finish_auth 0 2023-09-30T00:53:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae548a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae548a700 1 -- 172.21.15.40:0/3988675406 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad800de90 con 0x7f9ae013e2c0 2023-09-30T00:53:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae548a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae548a700 1 -- 172.21.15.40:0/3988675406 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad800b320 con 0x7f9ae013e2c0 2023-09-30T00:53:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae548a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae548a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae548a700 10 monclient: dump: 2023-09-30T00:53:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:10.785 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.785 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.785 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:10.785 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:10.785 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:10.785 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:10.785 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:10.785 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:10.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae86f0700 5 monclient: authenticate success, global_id 4915 2023-09-30T00:53:10.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae86f0700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:10.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.782+0000 7f9ae86f0700 4 set_mon_vals no callback set 2023-09-30T00:53:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.783+0000 7f9ae86f0700 10 monclient: shutdown 2023-09-30T00:53:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.783+0000 7f9ae86f0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.783+0000 7f9ae86f0700 1 -- 172.21.15.40:0/3988675406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ae013e2c0 msgr2=0x7f9ae0144f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.783+0000 7f9ae86f0700 1 --2- 172.21.15.40:0/3988675406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ae013e2c0 0x7f9ae0144f50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9ae013b690 tx=0x7f9ad800ba50 comp rx=0 tx=0).stop 2023-09-30T00:53:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.783+0000 7f9ae86f0700 1 -- 172.21.15.40:0/3988675406 shutdown_connections 2023-09-30T00:53:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.783+0000 7f9ae86f0700 1 --2- 172.21.15.40:0/3988675406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ae013e2c0 0x7f9ae0144f50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.783+0000 7f9ae86f0700 1 --2- 172.21.15.40:0/3988675406 >> v2:172.21.15.40:3300/0 conn(0x7f9ae006bd70 0x7f9ae006e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.783+0000 7f9ae86f0700 1 -- 172.21.15.40:0/3988675406 >> 172.21.15.40:0/3988675406 conn(0x7f9ae0067620 msgr2=0x7f9ae0067a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.783+0000 7f9ae86f0700 1 -- 172.21.15.40:0/3988675406 shutdown_connections 2023-09-30T00:53:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.783+0000 7f9ae86f0700 1 -- 172.21.15.40:0/3988675406 wait complete. 2023-09-30T00:53:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.784+0000 7f9ae86f0700 10 monclient: build_initial_monmap 2023-09-30T00:53:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.784+0000 7f9ae86f0700 10 monclient: monmap: 2023-09-30T00:53:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:10.791 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.784+0000 7f9ae86f0700 1 librados: starting msgr at 2023-09-30T00:53:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.784+0000 7f9ae86f0700 1 librados: starting objecter 2023-09-30T00:53:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.784+0000 7f9ae86f0700 1 Processor -- start 2023-09-30T00:53:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.784+0000 7f9ae86f0700 1 -- start start 2023-09-30T00:53:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.784+0000 7f9ae86f0700 1 librados: setting wanted keys 2023-09-30T00:53:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.784+0000 7f9ae86f0700 1 librados: calling monclient init 2023-09-30T00:53:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.784+0000 7f9ae86f0700 10 monclient: init 2023-09-30T00:53:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:53:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ae006bd70 0x7f9ae0140460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 10 monclient(hunting): picked mon.noname-c con 0x7f9ae006bd70 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ae006c140 0x7f9ae00b0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 10 monclient(hunting): picked mon.noname-b con 0x7f9ae006c140 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ae013e2c0 0x7f9ae0140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 10 monclient(hunting): picked mon.noname-a con 0x7f9ae013e2c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae0142920 con 0x7f9ae013e2c0 2023-09-30T00:53:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae0140ac0 con 0x7f9ae006c140 2023-09-30T00:53:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae00b0560 con 0x7f9ae006bd70 2023-09-30T00:53:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae86f0700 10 monclient(hunting): authenticate will time out at 5652.086246s 2023-09-30T00:53:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae6c8d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ae013e2c0 0x7f9ae0140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae5c8b700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ae006c140 0x7f9ae00b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:10.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.785+0000 7f9ae648c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ae006bd70 0x7f9ae0140460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:10.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae6c8d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ae013e2c0 0x7f9ae0140f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50020/0 (socket says 172.21.15.40:50020) 2023-09-30T00:53:10.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae6c8d700 1 -- 172.21.15.40:0/4183201815 learned_addr learned my addr 172.21.15.40:0/4183201815 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:10.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae6c8d700 10 monclient(hunting): get_auth_request con 0x7f9ae013e2c0 auth_method 0 2023-09-30T00:53:10.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae6c8d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:10.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae6c8d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:10.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae6c8d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:10.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae5c8b700 10 monclient(hunting): get_auth_request con 0x7f9ae006c140 auth_method 0 2023-09-30T00:53:10.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae5c8b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:10.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae5c8b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:10.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae5c8b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:10.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae648c700 10 monclient(hunting): get_auth_request con 0x7f9ae006bd70 auth_method 0 2023-09-30T00:53:10.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae648c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:10.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae648c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:10.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae648c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:10.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae5c8b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:10.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae5c8b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:10.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae5c8b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:10.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae6c8d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:10.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae6c8d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:10.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae6c8d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:10.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae648c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:10.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae648c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:10.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.786+0000 7f9ae648c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.789+0000 7f9ae5c8b700 10 monclient(hunting): handle_auth_done global_id 4918 payload 824 2023-09-30T00:53:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.789+0000 7f9ae5c8b700 1 -- 172.21.15.40:0/4183201815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ae006bd70 msgr2=0x7f9ae0140460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.789+0000 7f9ae5c8b700 1 --2- 172.21.15.40:0/4183201815 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9ae006bd70 0x7f9ae0140460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.789+0000 7f9ae5c8b700 1 -- 172.21.15.40:0/4183201815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ae013e2c0 msgr2=0x7f9ae0140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.789+0000 7f9ae5c8b700 1 --2- 172.21.15.40:0/4183201815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9ae013e2c0 0x7f9ae0140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.789+0000 7f9ae5c8b700 10 monclient: _finish_hunting 0 2023-09-30T00:53:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.789+0000 7f9ae5c8b700 1 monclient: found mon.noname-b 2023-09-30T00:53:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.789+0000 7f9ae5c8b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.789+0000 7f9ae5c8b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.789+0000 7f9ae5c8b700 1 -- 172.21.15.40:0/4183201815 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad800bcf0 con 0x7f9ae006c140 2023-09-30T00:53:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.792+0000 7f9ae5c8b700 1 --2- 172.21.15.40:0/4183201815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ae006c140 0x7f9ae00b0020 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000c5c0 tx=0x7f9ad0007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.792+0000 7f9ad67fc700 10 client.?.objecter ms_handle_connect 0x7f9ae006c140 2023-09-30T00:53:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.792+0000 7f9ad67fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.795+0000 7f9ad67fc700 1 -- 172.21.15.40:0/4183201815 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0011590 con 0x7f9ae006c140 2023-09-30T00:53:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.795+0000 7f9ad67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.795+0000 7f9ad67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.795+0000 7f9ad67fc700 10 monclient: dump: 2023-09-30T00:53:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:10.798 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:10.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.795+0000 7f9ad67fc700 10 monclient: _finish_auth 0 2023-09-30T00:53:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.795+0000 7f9ad67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.795+0000 7f9ad67fc700 1 -- 172.21.15.40:0/4183201815 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad0011710 con 0x7f9ae006c140 2023-09-30T00:53:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.795+0000 7f9ae86f0700 5 monclient: authenticate success, global_id 4918 2023-09-30T00:53:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.795+0000 7f9ae86f0700 10 monclient: _renew_subs 2023-09-30T00:53:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.795+0000 7f9ae86f0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.795+0000 7f9ae86f0700 1 -- 172.21.15.40:0/4183201815 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae00b0ef0 con 0x7f9ae006c140 2023-09-30T00:53:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.795+0000 7f9ad67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.795+0000 7f9ad67fc700 1 -- 172.21.15.40:0/4183201815 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0011ad0 con 0x7f9ae006c140 2023-09-30T00:53:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.795+0000 7f9ad67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.796+0000 7f9ad67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.796+0000 7f9ad67fc700 10 monclient: dump: 2023-09-30T00:53:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:10.800 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.800 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:10.800 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:10.800 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:10.800 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:10.800 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:10.800 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:10.800 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:10.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.796+0000 7f9ae4c89700 4 set_mon_vals no callback set 2023-09-30T00:53:10.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.796+0000 7f9ae86f0700 10 client.4918.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:10.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.796+0000 7f9ae86f0700 10 monclient: _renew_subs 2023-09-30T00:53:10.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.796+0000 7f9ae86f0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:10.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.796+0000 7f9ae86f0700 1 -- 172.21.15.40:0/4183201815 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae01416d0 con 0x7f9ae006c140 2023-09-30T00:53:10.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.796+0000 7f9ae86f0700 1 librados: init done 2023-09-30T00:53:10.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.797+0000 7f9ae86f0700 10 client.4918.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:10.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.797+0000 7f9ae86f0700 15 client.4918.objecter _session_command_op_assign -1 1 2023-09-30T00:53:10.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.797+0000 7f9ae86f0700 20 client.4918.objecter _get_session osd=-1 returning homeless 2023-09-30T00:53:10.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.797+0000 7f9ae86f0700 10 client.4918.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:10.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.797+0000 7f9ae86f0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:10.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.797+0000 7f9ae86f0700 1 -- 172.21.15.40:0/4183201815 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9ae0072b90 con 0x7f9ae006c140 2023-09-30T00:53:10.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.815+0000 7f9ad67fc700 1 -- 172.21.15.40:0/4183201815 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9ad001b3b0 con 0x7f9ae006c140 2023-09-30T00:53:10.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.816+0000 7f9ad67fc700 1 --2- 172.21.15.40:0/4183201815 >> v2:172.21.15.135:6800/116426 conn(0x7f9ac4068960 0x7f9ac406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:10.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.816+0000 7f9ae648c700 1 --2- 172.21.15.40:0/4183201815 >> v2:172.21.15.135:6800/116426 conn(0x7f9ac4068960 0x7f9ac406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:10.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.816+0000 7f9ae648c700 10 monclient: get_auth_request con 0x7f9ac4068960 auth_method 0 2023-09-30T00:53:10.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.816+0000 7f9ae648c700 1 --2- 172.21.15.40:0/4183201815 >> v2:172.21.15.135:6800/116426 conn(0x7f9ac4068960 0x7f9ac406ae10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9adc006fd0 tx=0x7f9adc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:10.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.817+0000 7f9ad67fc700 10 client.4918.objecter ms_handle_connect 0x7f9ac4068960 2023-09-30T00:53:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 1 -- 172.21.15.40:0/4183201815 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9ad0055600 con 0x7f9ae006c140 2023-09-30T00:53:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 10 client.4918.objecter ms_dispatch 0x7f9ae0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 3 client.4918.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 3 client.4918.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 10 client.4918.objecter checking command 1 2023-09-30T00:53:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 1 --2- 172.21.15.40:0/4183201815 >> v2:172.21.15.40:6800/117237 conn(0x7f9ac406e8d0 0x7f9ac4070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 20 client.4918.objecter _get_session s=0x7f9ac406e220 osd=1 3 2023-09-30T00:53:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 20 client.4918.objecter put_session s=0x7f9ac406e220 osd=1 3 2023-09-30T00:53:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 15 client.4918.objecter _session_command_op_remove -1 1 2023-09-30T00:53:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 20 client.4918.objecter _get_session s=0x7f9ac406e220 osd=1 3 2023-09-30T00:53:10.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 20 client.4918.objecter get_session s=0x7f9ac406e220 osd=1 3 2023-09-30T00:53:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 15 client.4918.objecter _session_command_op_assign 1 1 2023-09-30T00:53:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 20 client.4918.objecter put_session s=0x7f9ac406e220 osd=1 4 2023-09-30T00:53:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 10 client.4918.objecter _send_command 1 2023-09-30T00:53:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 1 -- 172.21.15.40:0/4183201815 --> v2:172.21.15.40:6800/117237 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ac4071350 con 0x7f9ac406e8d0 2023-09-30T00:53:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 20 client.4918.objecter dump_active .. 0 homeless 2023-09-30T00:53:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 1 -- 172.21.15.40:0/4183201815 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ad0001120 con 0x7f9ae006c140 2023-09-30T00:53:10.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ad67fc700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:53:10.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.821+0000 7f9ae6c8d700 1 --2- 172.21.15.40:0/4183201815 >> v2:172.21.15.40:6800/117237 conn(0x7f9ac406e8d0 0x7f9ac4070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:10.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.822+0000 7f9ae6c8d700 10 monclient: get_auth_request con 0x7f9ac406e8d0 auth_method 0 2023-09-30T00:53:10.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.825+0000 7f9ad67fc700 10 client.4918.objecter ms_handle_connect 0x7f9ac406e8d0 2023-09-30T00:53:10.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.825+0000 7f9ae6c8d700 1 --2- 172.21.15.40:0/4183201815 >> v2:172.21.15.40:6800/117237 conn(0x7f9ac406e8d0 0x7f9ac4070ce0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:10.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.850+0000 7f9ad67fc700 1 -- 172.21.15.40:0/4183201815 <== osd.1 v2:172.21.15.40:6800/117237 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f9ac4071350 con 0x7f9ac406e8d0 2023-09-30T00:53:10.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.850+0000 7f9ad67fc700 10 client.4918.objecter ms_dispatch 0x7f9ae0001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:53:10.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.850+0000 7f9ad67fc700 10 client.4918.objecter _finish_command 1 = system:0 2023-09-30T00:53:10.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.850+0000 7f9ad67fc700 20 client.4918.objecter put_session s=0x7f9ac406e220 osd=1 4 2023-09-30T00:53:10.851 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.850+0000 7f9ad67fc700 15 client.4918.objecter _session_command_op_remove 1 1 2023-09-30T00:53:10.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.865+0000 7f9ae86f0700 10 client.4918.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T00:53:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.865+0000 7f9ae86f0700 15 client.4918.objecter _session_command_op_assign -1 2 2023-09-30T00:53:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.865+0000 7f9ae86f0700 20 client.4918.objecter _get_session s=0x7f9ac406e220 osd=1 3 2023-09-30T00:53:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.865+0000 7f9ae86f0700 20 client.4918.objecter put_session s=0x7f9ac406e220 osd=1 3 2023-09-30T00:53:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.865+0000 7f9ae86f0700 20 client.4918.objecter _get_session s=0x7f9ac406e220 osd=1 3 2023-09-30T00:53:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.865+0000 7f9ae86f0700 15 client.4918.objecter _session_command_op_remove -1 2 2023-09-30T00:53:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.865+0000 7f9ae86f0700 20 client.4918.objecter get_session s=0x7f9ac406e220 osd=1 3 2023-09-30T00:53:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.865+0000 7f9ae86f0700 15 client.4918.objecter _session_command_op_assign 1 2 2023-09-30T00:53:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.865+0000 7f9ae86f0700 20 client.4918.objecter put_session s=0x7f9ac406e220 osd=1 4 2023-09-30T00:53:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.865+0000 7f9ae86f0700 10 client.4918.objecter _send_command 2 2023-09-30T00:53:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.865+0000 7f9ae86f0700 1 -- 172.21.15.40:0/4183201815 --> v2:172.21.15.40:6800/117237 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9ae006b250 con 0x7f9ac406e8d0 2023-09-30T00:53:10.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.889+0000 7f9ad67fc700 1 -- 172.21.15.40:0/4183201815 <== osd.1 v2:172.21.15.40:6800/117237 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f9ae006b250 con 0x7f9ac406e8d0 2023-09-30T00:53:10.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.889+0000 7f9ad67fc700 10 client.4918.objecter ms_dispatch 0x7f9ae0001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:53:10.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.889+0000 7f9ad67fc700 10 client.4918.objecter _finish_command 2 = system:0 2023-09-30T00:53:10.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.889+0000 7f9ad67fc700 20 client.4918.objecter put_session s=0x7f9ac406e220 osd=1 4 2023-09-30T00:53:10.890 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.889+0000 7f9ad67fc700 15 client.4918.objecter _session_command_op_remove 1 2 2023-09-30T00:53:10.890 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-09-30T00:53:10.890 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:53:10.891 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-09-30T00:53:10.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 10 librados: watch_flush enter 2023-09-30T00:53:10.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 10 librados: watch_flush exit 2023-09-30T00:53:10.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 10 client.4918.objecter close_session for osd.1 2023-09-30T00:53:10.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 1 -- 172.21.15.40:0/4183201815 >> v2:172.21.15.40:6800/117237 conn(0x7f9ac406e8d0 msgr2=0x7f9ac4070ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:10.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 1 --2- 172.21.15.40:0/4183201815 >> v2:172.21.15.40:6800/117237 conn(0x7f9ac406e8d0 0x7f9ac4070ce0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 20 client.4918.objecter put_session s=0x7f9ac406e220 osd=1 1 2023-09-30T00:53:10.892 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 20 client.4918.objecter shutdown clearing up homeless session... 2023-09-30T00:53:10.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 10 client.4918.objecter successfully canceled tick 2023-09-30T00:53:10.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 1 -- 172.21.15.40:0/4183201815 >> v2:172.21.15.135:6800/116426 conn(0x7f9ac4068960 msgr2=0x7f9ac406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:10.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 1 --2- 172.21.15.40:0/4183201815 >> v2:172.21.15.135:6800/116426 conn(0x7f9ac4068960 0x7f9ac406ae10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9adc006fd0 tx=0x7f9adc008040 comp rx=0 tx=0).stop 2023-09-30T00:53:10.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 10 monclient: shutdown 2023-09-30T00:53:10.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:10.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 1 -- 172.21.15.40:0/4183201815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ae006c140 msgr2=0x7f9ae00b0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:10.893 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 1 --2- 172.21.15.40:0/4183201815 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9ae006c140 0x7f9ae00b0020 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000c5c0 tx=0x7f9ad0007ec0 comp rx=0 tx=0).stop 2023-09-30T00:53:10.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae6c8d700 1 -- 172.21.15.40:0/4183201815 reap_dead start 2023-09-30T00:53:10.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 1 -- 172.21.15.40:0/4183201815 shutdown_connections 2023-09-30T00:53:10.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 1 -- 172.21.15.40:0/4183201815 >> 172.21.15.40:0/4183201815 conn(0x7f9ae0067620 msgr2=0x7f9ae00a98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:10.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 1 -- 172.21.15.40:0/4183201815 shutdown_connections 2023-09-30T00:53:10.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 1 -- 172.21.15.40:0/4183201815 wait complete. 2023-09-30T00:53:10.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:10.890+0000 7f9ae86f0700 1 librados: shutdown 2023-09-30T00:53:10.904 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-30T00:53:11.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.037+0000 7f8bdebf8700 10 monclient: get_monmap_and_config 2023-09-30T00:53:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.037+0000 7f8bdebf8700 10 monclient: build_initial_monmap 2023-09-30T00:53:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.037+0000 7f8bdebf8700 10 monclient: monmap: 2023-09-30T00:53:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:11.038777+0000 2023-09-30T00:53:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:11.038777+0000 2023-09-30T00:53:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:11.040 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.038+0000 7f8bdebf8700 1 Processor -- start 2023-09-30T00:53:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.038+0000 7f8bdebf8700 1 -- start start 2023-09-30T00:53:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.038+0000 7f8bdebf8700 10 monclient: init 2023-09-30T00:53:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.039+0000 7f8bdebf8700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.039+0000 7f8bdebf8700 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-09-30T00:53:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.039+0000 7f8bdebf8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8bd80b1460 0x7f8bd80b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.039+0000 7f8bdebf8700 10 monclient(hunting): picked mon.noname-a con 0x7f8bd80b1460 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.039+0000 7f8bdebf8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8bd80a5fb0 addr v1:172.21.15.40:6789/0 2023-09-30T00:53:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.039+0000 7f8bdebf8700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8bd813ac20 0x7f8bd813b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.039+0000 7f8bdebf8700 10 monclient(hunting): picked mon.noname-c con 0x7f8bd813ac20 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.039+0000 7f8bdebf8700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd80a5b20 con 0x7f8bd80a5fb0 2023-09-30T00:53:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.039+0000 7f8bdebf8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.039+0000 7f8bdebf8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd80a71f0 con 0x7f8bd80b1460 2023-09-30T00:53:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.039+0000 7f8bdebf8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.039+0000 7f8bdebf8700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd80a74d0 con 0x7f8bd813ac20 2023-09-30T00:53:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.039+0000 7f8bdebf8700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.039+0000 7f8bdebf8700 10 monclient(hunting): authenticate will time out at 5652.340306s 2023-09-30T00:53:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.040+0000 7f8bdd195700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f8bd813ac20 0x7f8bd813b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:11.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.040+0000 7f8bdc994700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8bd80b1460 0x7f8bd80b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:11.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.040+0000 7f8bd7fff700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8bd80a5fb0 0x7f8bd8141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:54398/0 (socket says 172.21.15.40:54398) 2023-09-30T00:53:11.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.040+0000 7f8bd7fff700 1 -- 172.21.15.40:0/2619259126 learned_addr learned my addr 172.21.15.40:0/2619259126 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:53:11.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.040+0000 7f8bdd195700 10 monclient(hunting): get_auth_request con 0x7f8bd813ac20 auth_method 0 2023-09-30T00:53:11.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.040+0000 7f8bdd195700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:11.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.040+0000 7f8bdd195700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:11.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.040+0000 7f8bdd195700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:11.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.040+0000 7f8bdc994700 10 monclient(hunting): get_auth_request con 0x7f8bd80b1460 auth_method 0 2023-09-30T00:53:11.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.040+0000 7f8bdc994700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.040+0000 7f8bdc994700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.040+0000 7f8bdc994700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.041+0000 7f8bdd195700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.041+0000 7f8bdd195700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.041+0000 7f8bdd195700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.041+0000 7f8bdc994700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.041+0000 7f8bdc994700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.041+0000 7f8bdc994700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.043+0000 7f8bdd195700 10 monclient(hunting): handle_auth_done global_id 4921 payload 274 2023-09-30T00:53:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.043+0000 7f8bdd195700 1 -- 172.21.15.40:0/2619259126 >> v2:172.21.15.40:3300/0 conn(0x7f8bd80b1460 msgr2=0x7f8bd80b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.043+0000 7f8bdd195700 1 --2- 172.21.15.40:0/2619259126 >> v2:172.21.15.40:3300/0 conn(0x7f8bd80b1460 0x7f8bd80b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.043+0000 7f8bdd195700 1 -- 172.21.15.40:0/2619259126 >> v1:172.21.15.40:6789/0 conn(0x7f8bd80a5fb0 legacy=0x7f8bd8141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.043+0000 7f8bdd195700 10 monclient: _finish_hunting 0 2023-09-30T00:53:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.043+0000 7f8bdd195700 1 monclient: found mon.noname-c 2023-09-30T00:53:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.043+0000 7f8bdd195700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.043+0000 7f8bdd195700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:53:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.043+0000 7f8bdd195700 1 -- 172.21.15.40:0/2619259126 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd80a63c0 con 0x7f8bd813ac20 2023-09-30T00:53:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.045+0000 7f8bdd195700 1 --2- 172.21.15.40:0/2619259126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd813ac20 0x7f8bd813b080 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0009e60 tx=0x7f8bd0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8629e198b68e94be server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:11.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bd77fe700 1 -- 172.21.15.40:0/2619259126 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd000e070 con 0x7f8bd813ac20 2023-09-30T00:53:11.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bd77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:11.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bd77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:53:11.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bd77fe700 10 monclient: dump: 2023-09-30T00:53:11.050 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:11.050 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:11.050 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.050 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.050 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:11.050 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:11.050 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:11.050 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:11.050 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:11.050 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bd77fe700 10 monclient: _finish_auth 0 2023-09-30T00:53:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bd77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bd77fe700 1 -- 172.21.15.40:0/2619259126 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd000bb20 con 0x7f8bd813ac20 2023-09-30T00:53:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bdebf8700 5 monclient: authenticate success, global_id 4921 2023-09-30T00:53:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bdebf8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bd77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bdebf8700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bd77fe700 1 -- 172.21.15.40:0/2619259126 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd0012430 con 0x7f8bd813ac20 2023-09-30T00:53:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bdebf8700 4 set_mon_vals no callback set 2023-09-30T00:53:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bd77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:11.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bd77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:11.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bd77fe700 10 monclient: dump: 2023-09-30T00:53:11.052 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:11.052 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:11.052 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.052 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.052 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:11.052 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:11.052 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:11.052 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:11.053 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:11.053 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:11.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bdebf8700 10 monclient: shutdown 2023-09-30T00:53:11.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bdebf8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:11.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2619259126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd813ac20 msgr2=0x7f8bd813b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:11.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.049+0000 7f8bdebf8700 1 --2- 172.21.15.40:0/2619259126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd813ac20 0x7f8bd813b080 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0009e60 tx=0x7f8bd0009780 comp rx=0 tx=0).stop 2023-09-30T00:53:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.050+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2619259126 shutdown_connections 2023-09-30T00:53:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.050+0000 7f8bdebf8700 1 --2- 172.21.15.40:0/2619259126 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd813ac20 0x7f8bd813b080 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.050+0000 7f8bdebf8700 1 --2- 172.21.15.40:0/2619259126 >> v2:172.21.15.40:3300/0 conn(0x7f8bd80b1460 0x7f8bd80b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.050+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2619259126 >> 172.21.15.40:0/2619259126 conn(0x7f8bd80631f0 msgr2=0x7f8bd80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.050+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2619259126 shutdown_connections 2023-09-30T00:53:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.050+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2619259126 wait complete. 2023-09-30T00:53:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.050+0000 7f8bdebf8700 10 monclient: build_initial_monmap 2023-09-30T00:53:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.050+0000 7f8bdebf8700 10 monclient: monmap: 2023-09-30T00:53:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:11.060 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.050+0000 7f8bdebf8700 1 librados: starting msgr at 2023-09-30T00:53:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.050+0000 7f8bdebf8700 1 librados: starting objecter 2023-09-30T00:53:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 1 Processor -- start 2023-09-30T00:53:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 1 -- start start 2023-09-30T00:53:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 1 librados: setting wanted keys 2023-09-30T00:53:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 1 librados: calling monclient init 2023-09-30T00:53:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 10 monclient: init 2023-09-30T00:53:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:53:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd80a5fb0 0x7f8bd813dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 10 monclient(hunting): picked mon.noname-b con 0x7f8bd80a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bd80b1460 0x7f8bd80a9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 10 monclient(hunting): picked mon.noname-a con 0x7f8bd80b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bd813ac20 0x7f8bd814d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 10 monclient(hunting): picked mon.noname-c con 0x7f8bd813ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd80ab410 con 0x7f8bd80b1460 2023-09-30T00:53:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd80a98d0 con 0x7f8bd80a5fb0 2023-09-30T00:53:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd814b2d0 con 0x7f8bd813ac20 2023-09-30T00:53:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.051+0000 7f8bdebf8700 10 monclient(hunting): authenticate will time out at 5652.352964s 2023-09-30T00:53:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bd7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bd80b1460 0x7f8bd80a9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bdd195700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bd813ac20 0x7f8bd814d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bdc994700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd80a5fb0 0x7f8bd813dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bd7fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bd80b1460 0x7f8bd80a9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50034/0 (socket says 172.21.15.40:50034) 2023-09-30T00:53:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bd7fff700 1 -- 172.21.15.40:0/2206176843 learned_addr learned my addr 172.21.15.40:0/2206176843 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bd7fff700 10 monclient(hunting): get_auth_request con 0x7f8bd80b1460 auth_method 0 2023-09-30T00:53:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bd7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bd7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:11.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bd7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bdc994700 10 monclient(hunting): get_auth_request con 0x7f8bd80a5fb0 auth_method 0 2023-09-30T00:53:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bdc994700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bdc994700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bdc994700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bdd195700 10 monclient(hunting): get_auth_request con 0x7f8bd813ac20 auth_method 0 2023-09-30T00:53:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.052+0000 7f8bdd195700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.053+0000 7f8bdd195700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.053+0000 7f8bdd195700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.053+0000 7f8bdc994700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.053+0000 7f8bdc994700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.053+0000 7f8bdc994700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:11.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.053+0000 7f8bd7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.053+0000 7f8bd7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.053+0000 7f8bd7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.053+0000 7f8bdd195700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.053+0000 7f8bdd195700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.053+0000 7f8bdd195700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.056+0000 7f8bdc994700 10 monclient(hunting): handle_auth_done global_id 4924 payload 824 2023-09-30T00:53:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.056+0000 7f8bdc994700 1 -- 172.21.15.40:0/2206176843 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bd813ac20 msgr2=0x7f8bd814d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.056+0000 7f8bdc994700 1 --2- 172.21.15.40:0/2206176843 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bd813ac20 0x7f8bd814d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.056+0000 7f8bdc994700 1 -- 172.21.15.40:0/2206176843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bd80b1460 msgr2=0x7f8bd80a9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.056+0000 7f8bdc994700 1 --2- 172.21.15.40:0/2206176843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bd80b1460 0x7f8bd80a9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.056+0000 7f8bdc994700 10 monclient: _finish_hunting 0 2023-09-30T00:53:11.065 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.056+0000 7f8bdc994700 1 monclient: found mon.noname-b 2023-09-30T00:53:11.065 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.056+0000 7f8bdc994700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:11.065 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.056+0000 7f8bdc994700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:11.065 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.056+0000 7f8bdc994700 1 -- 172.21.15.40:0/2206176843 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd0009a20 con 0x7f8bd80a5fb0 2023-09-30T00:53:11.066 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.058+0000 7f8bdc994700 1 --2- 172.21.15.40:0/2206176843 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd80a5fb0 0x7f8bd813dd30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800ea50 tx=0x7f8bc80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:11.066 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.058+0000 7f8bd4ff9700 10 client.?.objecter ms_handle_connect 0x7f8bd80a5fb0 2023-09-30T00:53:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.058+0000 7f8bd4ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bd4ff9700 1 -- 172.21.15.40:0/2206176843 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc800ee90 con 0x7f8bd80a5fb0 2023-09-30T00:53:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bd4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bd4ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bd4ff9700 10 monclient: dump: 2023-09-30T00:53:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:11.068 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bd4ff9700 10 monclient: _finish_auth 0 2023-09-30T00:53:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bd4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bd4ff9700 1 -- 172.21.15.40:0/2206176843 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc800c7f0 con 0x7f8bd80a5fb0 2023-09-30T00:53:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bd4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bd4ff9700 1 -- 172.21.15.40:0/2206176843 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc800cb90 con 0x7f8bd80a5fb0 2023-09-30T00:53:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bd4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:11.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bd4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bd4ff9700 10 monclient: dump: 2023-09-30T00:53:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:11.069 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:11.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bdebf8700 5 monclient: authenticate success, global_id 4924 2023-09-30T00:53:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bdebf8700 10 monclient: _renew_subs 2023-09-30T00:53:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bdebf8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bd6ffd700 4 set_mon_vals no callback set 2023-09-30T00:53:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2206176843 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd813e920 con 0x7f8bd80a5fb0 2023-09-30T00:53:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bdebf8700 10 client.4924.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bdebf8700 10 monclient: _renew_subs 2023-09-30T00:53:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bdebf8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2206176843 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd80a9600 con 0x7f8bd80a5fb0 2023-09-30T00:53:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.064+0000 7f8bdebf8700 1 librados: init done 2023-09-30T00:53:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.065+0000 7f8bdebf8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:11.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.065+0000 7f8bdebf8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:11.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.065+0000 7f8bdebf8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:11.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.065+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2206176843 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd80a9a90 con 0x7f8bd80a5fb0 2023-09-30T00:53:11.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.087+0000 7f8bd4ff9700 1 -- 172.21.15.40:0/2206176843 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8bc8013470 con 0x7f8bd80a5fb0 2023-09-30T00:53:11.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.087+0000 7f8bd4ff9700 1 --2- 172.21.15.40:0/2206176843 >> v2:172.21.15.135:6800/116426 conn(0x7f8bb8068890 0x7f8bb806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:11.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.087+0000 7f8bd7fff700 1 --2- 172.21.15.40:0/2206176843 >> v2:172.21.15.135:6800/116426 conn(0x7f8bb8068890 0x7f8bb806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:11.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.087+0000 7f8bd7fff700 10 monclient: get_auth_request con 0x7f8bb8068890 auth_method 0 2023-09-30T00:53:11.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.088+0000 7f8bd7fff700 1 --2- 172.21.15.40:0/2206176843 >> v2:172.21.15.135:6800/116426 conn(0x7f8bb8068890 0x7f8bb806ad40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8bd813dab0 tx=0x7f8bc0007f70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:11.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.088+0000 7f8bd4ff9700 10 client.4924.objecter ms_handle_connect 0x7f8bb8068890 2023-09-30T00:53:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.092+0000 7f8bd4ff9700 1 -- 172.21.15.40:0/2206176843 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8bc80564c0 con 0x7f8bd80a5fb0 2023-09-30T00:53:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.092+0000 7f8bd4ff9700 10 client.4924.objecter ms_dispatch 0x7f8bd8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.092+0000 7f8bd4ff9700 3 client.4924.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.092+0000 7f8bd4ff9700 3 client.4924.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:11.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.092+0000 7f8bd4ff9700 20 client.4924.objecter dump_active .. 0 homeless 2023-09-30T00:53:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.355+0000 7f8bd4ff9700 1 -- 172.21.15.40:0/2206176843 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8bc8052100 con 0x7f8bd80a5fb0 2023-09-30T00:53:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.355+0000 7f8bd4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.355+0000 7f8bd4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:11.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.564+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2206176843 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8bd8064a00 con 0x7f8bb8068890 2023-09-30T00:53:11.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.564+0000 7f8bd4ff9700 1 -- 172.21.15.40:0/2206176843 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8bd8064a00 con 0x7f8bb8068890 2023-09-30T00:53:11.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.566+0000 7f8bdebf8700 10 librados: watch_flush enter 2023-09-30T00:53:11.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.566+0000 7f8bdebf8700 10 librados: watch_flush exit 2023-09-30T00:53:11.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.566+0000 7f8bdebf8700 20 client.4924.objecter shutdown clearing up homeless session... 2023-09-30T00:53:11.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.566+0000 7f8bdebf8700 10 client.4924.objecter successfully canceled tick 2023-09-30T00:53:11.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.566+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2206176843 >> v2:172.21.15.135:6800/116426 conn(0x7f8bb8068890 msgr2=0x7f8bb806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:11.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.567+0000 7f8bdebf8700 1 --2- 172.21.15.40:0/2206176843 >> v2:172.21.15.135:6800/116426 conn(0x7f8bb8068890 0x7f8bb806ad40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8bd813dab0 tx=0x7f8bc0007f70 comp rx=0 tx=0).stop 2023-09-30T00:53:11.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.567+0000 7f8bdebf8700 10 monclient: shutdown 2023-09-30T00:53:11.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.567+0000 7f8bdebf8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:11.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.567+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2206176843 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd80a5fb0 msgr2=0x7f8bd813dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:11.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.567+0000 7f8bdebf8700 1 --2- 172.21.15.40:0/2206176843 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd80a5fb0 0x7f8bd813dd30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800ea50 tx=0x7f8bc80091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.567+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2206176843 shutdown_connections 2023-09-30T00:53:11.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.567+0000 7f8bdebf8700 1 --2- 172.21.15.40:0/2206176843 >> v2:172.21.15.135:6800/116426 conn(0x7f8bb8068890 0x7f8bb806ad40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.567+0000 7f8bdebf8700 1 --2- 172.21.15.40:0/2206176843 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f8bd80a5fb0 0x7f8bd813dd30 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.567+0000 7f8bdebf8700 1 --2- 172.21.15.40:0/2206176843 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f8bd813ac20 0x7f8bd814d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.567+0000 7f8bdebf8700 1 --2- 172.21.15.40:0/2206176843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bd80b1460 0x7f8bd80a9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.567+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2206176843 >> 172.21.15.40:0/2206176843 conn(0x7f8bd80631f0 msgr2=0x7f8bd80af310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:11.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.567+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2206176843 shutdown_connections 2023-09-30T00:53:11.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.567+0000 7f8bdebf8700 1 -- 172.21.15.40:0/2206176843 wait complete. 2023-09-30T00:53:11.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.567+0000 7f8bdebf8700 1 librados: shutdown 2023-09-30T00:53:11.570 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 1 to deep-scrub 2023-09-30T00:53:11.581 INFO:tasks.ceph:Scrubbing osd.2 2023-09-30T00:53:11.581 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:53:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.723+0000 7fe98ba3c700 10 monclient: get_monmap_and_config 2023-09-30T00:53:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.723+0000 7fe98ba3c700 10 monclient: build_initial_monmap 2023-09-30T00:53:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.723+0000 7fe98ba3c700 10 monclient: monmap: 2023-09-30T00:53:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:11.724427+0000 2023-09-30T00:53:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:11.724427+0000 2023-09-30T00:53:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:11.725 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:11.725 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:11.725 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:11.725 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:11.725 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:11.725 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:11.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 1 Processor -- start 2023-09-30T00:53:11.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 1 -- start start 2023-09-30T00:53:11.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 10 monclient: init 2023-09-30T00:53:11.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:11.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-09-30T00:53:11.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9840a5710 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:11.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe9840a7630 0x7fe9840a7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:11.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 10 monclient(hunting): picked mon.noname-c con 0x7fe9840a7630 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:11.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9840a7fc0 0x7fe98413af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:11.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 10 monclient(hunting): picked mon.noname-b con 0x7fe9840a7fc0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:11.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9840b1770 con 0x7fe9840a5710 2023-09-30T00:53:11.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:11.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9840a5ae0 con 0x7fe9840a7630 2023-09-30T00:53:11.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:11.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9840b1440 con 0x7fe9840a7fc0 2023-09-30T00:53:11.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:11.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.724+0000 7fe98ba3c700 10 monclient(hunting): authenticate will time out at 5653.025986s 2023-09-30T00:53:11.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.754+0000 7fe988fd7700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe9840a7630 0x7fe9840a7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:11.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.755+0000 7fe988fd7700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fe9840a7630 0x7fe9840a7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:56476/0 (socket says 172.21.15.40:56476) 2023-09-30T00:53:11.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.755+0000 7fe988fd7700 1 -- 172.21.15.40:0/3355665799 learned_addr learned my addr 172.21.15.40:0/3355665799 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:11.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.755+0000 7fe988fd7700 10 monclient(hunting): get_auth_request con 0x7fe9840a7630 auth_method 0 2023-09-30T00:53:11.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.755+0000 7fe988fd7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:11.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.755+0000 7fe988fd7700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:11.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.755+0000 7fe988fd7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:11.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.756+0000 7fe988fd7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:11.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.756+0000 7fe988fd7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:11.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.756+0000 7fe988fd7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:11.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.758+0000 7fe988fd7700 10 monclient(hunting): handle_auth_done global_id 4927 payload 274 2023-09-30T00:53:11.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.758+0000 7fe988fd7700 1 -- 172.21.15.40:0/3355665799 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9840a7fc0 msgr2=0x7fe98413af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:11.758 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.758+0000 7fe988fd7700 1 --2- 172.21.15.40:0/3355665799 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9840a7fc0 0x7fe98413af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.758+0000 7fe988fd7700 1 -- 172.21.15.40:0/3355665799 >> v1:172.21.15.135:6789/0 conn(0x7fe9840a5710 legacy=0x7fe9840b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:11.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.758+0000 7fe988fd7700 10 monclient: _finish_hunting 0 2023-09-30T00:53:11.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.758+0000 7fe988fd7700 1 monclient: found mon.noname-c 2023-09-30T00:53:11.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.758+0000 7fe988fd7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:11.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.758+0000 7fe988fd7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:53:11.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.758+0000 7fe988fd7700 1 -- 172.21.15.40:0/3355665799 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9840a7210 con 0x7fe9840a7630 2023-09-30T00:53:11.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.760+0000 7fe988fd7700 1 --2- 172.21.15.40:0/3355665799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9840a7630 0x7fe9840a7a80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe9740099b0 tx=0x7fe974009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c326addb3433f38 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:11.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe97bfff700 1 -- 172.21.15.40:0/3355665799 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97400b820 con 0x7fe9840a7630 2023-09-30T00:53:11.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe97bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:11.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe97bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:53:11.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe97bfff700 10 monclient: dump: 2023-09-30T00:53:11.768 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:11.768 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:11.768 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.768 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.768 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:11.768 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:11.768 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:11.769 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:11.769 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:11.769 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:11.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe97bfff700 10 monclient: _finish_auth 0 2023-09-30T00:53:11.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe97bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:11.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe97bfff700 1 -- 172.21.15.40:0/3355665799 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97400b980 con 0x7fe9840a7630 2023-09-30T00:53:11.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe98ba3c700 5 monclient: authenticate success, global_id 4927 2023-09-30T00:53:11.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe98ba3c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:11.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe97bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:11.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe98ba3c700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:11.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe98ba3c700 4 set_mon_vals no callback set 2023-09-30T00:53:11.769 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe97bfff700 1 -- 172.21.15.40:0/3355665799 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97400bd20 con 0x7fe9840a7630 2023-09-30T00:53:11.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe97bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:11.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe97bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:11.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.767+0000 7fe97bfff700 10 monclient: dump: 2023-09-30T00:53:11.770 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:11.770 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:11.770 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.770 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.770 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:11.770 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:11.770 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:11.770 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:11.770 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:11.771 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:11.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.768+0000 7fe98ba3c700 10 monclient: shutdown 2023-09-30T00:53:11.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.768+0000 7fe98ba3c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:11.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.768+0000 7fe98ba3c700 1 -- 172.21.15.40:0/3355665799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9840a7630 msgr2=0x7fe9840a7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:11.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.768+0000 7fe98ba3c700 1 --2- 172.21.15.40:0/3355665799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9840a7630 0x7fe9840a7a80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe9740099b0 tx=0x7fe974009250 comp rx=0 tx=0).stop 2023-09-30T00:53:11.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.768+0000 7fe98ba3c700 1 -- 172.21.15.40:0/3355665799 shutdown_connections 2023-09-30T00:53:11.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.768+0000 7fe98ba3c700 1 --2- 172.21.15.40:0/3355665799 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9840a7fc0 0x7fe98413af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.768+0000 7fe98ba3c700 1 --2- 172.21.15.40:0/3355665799 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9840a7630 0x7fe9840a7a80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.768+0000 7fe98ba3c700 1 -- 172.21.15.40:0/3355665799 >> 172.21.15.40:0/3355665799 conn(0x7fe984069060 msgr2=0x7fe9840631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:11.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.768+0000 7fe98ba3c700 1 -- 172.21.15.40:0/3355665799 shutdown_connections 2023-09-30T00:53:11.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.768+0000 7fe98ba3c700 1 -- 172.21.15.40:0/3355665799 wait complete. 2023-09-30T00:53:11.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.769+0000 7fe98ba3c700 10 monclient: build_initial_monmap 2023-09-30T00:53:11.775 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.769+0000 7fe98ba3c700 10 monclient: monmap: 2023-09-30T00:53:11.775 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:11.775 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:11.775 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:11.775 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:11.776 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:11.776 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:11.776 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:11.776 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:11.776 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:11.776 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:11.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.769+0000 7fe98ba3c700 1 librados: starting msgr at 2023-09-30T00:53:11.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.769+0000 7fe98ba3c700 1 librados: starting objecter 2023-09-30T00:53:11.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.769+0000 7fe98ba3c700 1 Processor -- start 2023-09-30T00:53:11.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.769+0000 7fe98ba3c700 1 -- start start 2023-09-30T00:53:11.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.769+0000 7fe98ba3c700 1 librados: setting wanted keys 2023-09-30T00:53:11.776 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.769+0000 7fe98ba3c700 1 librados: calling monclient init 2023-09-30T00:53:11.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.769+0000 7fe98ba3c700 10 monclient: init 2023-09-30T00:53:11.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:11.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:53:11.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9840a5710 0x7fe9841519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:11.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 10 monclient(hunting): picked mon.noname-a con 0x7fe9840a5710 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:11.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9840a7630 0x7fe984150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:11.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 10 monclient(hunting): picked mon.noname-c con 0x7fe9840a7630 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:11.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9840a7fc0 0x7fe984140af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:11.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 10 monclient(hunting): picked mon.noname-b con 0x7fe9840a7fc0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:11.777 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:11.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe984151550 con 0x7fe9840a5710 2023-09-30T00:53:11.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:11.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9840aa660 con 0x7fe9840a7fc0 2023-09-30T00:53:11.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:11.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe984144580 con 0x7fe9840a7630 2023-09-30T00:53:11.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:11.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.770+0000 7fe98ba3c700 10 monclient(hunting): authenticate will time out at 5653.071756s 2023-09-30T00:53:11.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.771+0000 7fe9897d8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9840a5710 0x7fe9841519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:11.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.771+0000 7fe989fd9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9840a7fc0 0x7fe984140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:11.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.771+0000 7fe989fd9700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9840a7fc0 0x7fe984140af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:56478/0 (socket says 172.21.15.40:56478) 2023-09-30T00:53:11.778 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.771+0000 7fe989fd9700 1 -- 172.21.15.40:0/1548417246 learned_addr learned my addr 172.21.15.40:0/1548417246 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:11.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.771+0000 7fe989fd9700 10 monclient(hunting): get_auth_request con 0x7fe9840a7fc0 auth_method 0 2023-09-30T00:53:11.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.771+0000 7fe989fd9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:11.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.772+0000 7fe989fd9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:11.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.772+0000 7fe989fd9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:11.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.772+0000 7fe9897d8700 10 monclient(hunting): get_auth_request con 0x7fe9840a5710 auth_method 0 2023-09-30T00:53:11.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.772+0000 7fe9897d8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:11.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.772+0000 7fe9897d8700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:11.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.772+0000 7fe9897d8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:11.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.772+0000 7fe989fd9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:11.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.772+0000 7fe989fd9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:11.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.772+0000 7fe989fd9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:11.779 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.772+0000 7fe9897d8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:11.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.772+0000 7fe9897d8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:11.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.772+0000 7fe9897d8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:11.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.772+0000 7fe988fd7700 1 --2- 172.21.15.40:0/1548417246 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9840a7630 0x7fe984150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:11.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.773+0000 7fe988fd7700 10 monclient(hunting): get_auth_request con 0x7fe9840a7630 auth_method 0 2023-09-30T00:53:11.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.773+0000 7fe988fd7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:11.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.773+0000 7fe988fd7700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:11.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.773+0000 7fe988fd7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:11.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.773+0000 7fe988fd7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:11.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.773+0000 7fe988fd7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:11.780 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.773+0000 7fe988fd7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:11.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.775+0000 7fe989fd9700 10 monclient(hunting): handle_auth_done global_id 4930 payload 824 2023-09-30T00:53:11.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.775+0000 7fe989fd9700 1 -- 172.21.15.40:0/1548417246 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9840a7630 msgr2=0x7fe984150ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:11.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.775+0000 7fe989fd9700 1 --2- 172.21.15.40:0/1548417246 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fe9840a7630 0x7fe984150ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.775+0000 7fe989fd9700 1 -- 172.21.15.40:0/1548417246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9840a5710 msgr2=0x7fe9841519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:11.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.775+0000 7fe989fd9700 1 --2- 172.21.15.40:0/1548417246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe9840a5710 0x7fe9841519e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.775+0000 7fe989fd9700 10 monclient: _finish_hunting 0 2023-09-30T00:53:11.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.775+0000 7fe989fd9700 1 monclient: found mon.noname-b 2023-09-30T00:53:11.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.775+0000 7fe989fd9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:11.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.775+0000 7fe989fd9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:11.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.775+0000 7fe989fd9700 1 -- 172.21.15.40:0/1548417246 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9740094f0 con 0x7fe9840a7fc0 2023-09-30T00:53:11.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.777+0000 7fe989fd9700 1 --2- 172.21.15.40:0/1548417246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9840a7fc0 0x7fe984140af0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00b5e0 tx=0x7fe97c009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:11.783 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.777+0000 7fe9797fa700 10 client.?.objecter ms_handle_connect 0x7fe9840a7fc0 2023-09-30T00:53:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.777+0000 7fe9797fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe9797fa700 1 -- 172.21.15.40:0/1548417246 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c00f510 con 0x7fe9840a7fc0 2023-09-30T00:53:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe9797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe9797fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe9797fa700 10 monclient: dump: 2023-09-30T00:53:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:11.785 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:11.785 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:11.785 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:11.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe9797fa700 10 monclient: _finish_auth 0 2023-09-30T00:53:11.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe9797fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:11.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe9797fa700 1 -- 172.21.15.40:0/1548417246 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97c00f670 con 0x7fe9840a7fc0 2023-09-30T00:53:11.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe98ba3c700 5 monclient: authenticate success, global_id 4930 2023-09-30T00:53:11.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe9797fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:11.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe9797fa700 1 -- 172.21.15.40:0/1548417246 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c00fa30 con 0x7fe9840a7fc0 2023-09-30T00:53:11.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe98ba3c700 10 monclient: _renew_subs 2023-09-30T00:53:11.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe98ba3c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:11.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe98ba3c700 1 -- 172.21.15.40:0/1548417246 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9840aa8d0 con 0x7fe9840a7fc0 2023-09-30T00:53:11.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe9797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:11.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe9797fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:11.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe97b7fe700 4 set_mon_vals no callback set 2023-09-30T00:53:11.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe9797fa700 10 monclient: dump: 2023-09-30T00:53:11.786 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:11.786 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:11.786 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.786 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:11.786 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:11.786 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:11.787 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:11.787 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:11.787 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:11.787 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:11.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe98ba3c700 10 client.4930.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:11.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe98ba3c700 10 monclient: _renew_subs 2023-09-30T00:53:11.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe98ba3c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:11.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe98ba3c700 1 -- 172.21.15.40:0/1548417246 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9840a87d0 con 0x7fe9840a7fc0 2023-09-30T00:53:11.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.781+0000 7fe98ba3c700 1 librados: init done 2023-09-30T00:53:11.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.782+0000 7fe98ba3c700 10 client.4930.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:11.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.782+0000 7fe98ba3c700 15 client.4930.objecter _session_command_op_assign -1 1 2023-09-30T00:53:11.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.782+0000 7fe98ba3c700 20 client.4930.objecter _get_session osd=-1 returning homeless 2023-09-30T00:53:11.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.782+0000 7fe98ba3c700 10 client.4930.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:11.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.782+0000 7fe98ba3c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:11.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.782+0000 7fe98ba3c700 1 -- 172.21.15.40:0/1548417246 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe984064a00 con 0x7fe9840a7fc0 2023-09-30T00:53:11.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.801+0000 7fe9797fa700 1 -- 172.21.15.40:0/1548417246 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fe97c024430 con 0x7fe9840a7fc0 2023-09-30T00:53:11.802 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.802+0000 7fe9797fa700 1 --2- 172.21.15.40:0/1548417246 >> v2:172.21.15.135:6800/116426 conn(0x7fe968068960 0x7fe96806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:11.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.802+0000 7fe9897d8700 1 --2- 172.21.15.40:0/1548417246 >> v2:172.21.15.135:6800/116426 conn(0x7fe968068960 0x7fe96806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:11.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.802+0000 7fe9897d8700 10 monclient: get_auth_request con 0x7fe968068960 auth_method 0 2023-09-30T00:53:11.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.803+0000 7fe9797fa700 10 client.4930.objecter ms_handle_connect 0x7fe968068960 2023-09-30T00:53:11.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.803+0000 7fe9897d8700 1 --2- 172.21.15.40:0/1548417246 >> v2:172.21.15.135:6800/116426 conn(0x7fe968068960 0x7fe96806ae10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe980007fd0 tx=0x7fe9800026a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:11.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 1 -- 172.21.15.40:0/1548417246 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fe97c055f60 con 0x7fe9840a7fc0 2023-09-30T00:53:11.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 10 client.4930.objecter ms_dispatch 0x7fe984001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:11.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 3 client.4930.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:11.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 3 client.4930.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:11.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 10 client.4930.objecter checking command 1 2023-09-30T00:53:11.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 1 --2- 172.21.15.40:0/1548417246 >> v2:172.21.15.40:6808/117239 conn(0x7fe96806e8d0 0x7fe968070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:11.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 20 client.4930.objecter _get_session s=0x7fe96806e220 osd=2 3 2023-09-30T00:53:11.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 20 client.4930.objecter put_session s=0x7fe96806e220 osd=2 3 2023-09-30T00:53:11.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 15 client.4930.objecter _session_command_op_remove -1 1 2023-09-30T00:53:11.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 20 client.4930.objecter _get_session s=0x7fe96806e220 osd=2 3 2023-09-30T00:53:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 20 client.4930.objecter get_session s=0x7fe96806e220 osd=2 3 2023-09-30T00:53:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 15 client.4930.objecter _session_command_op_assign 2 1 2023-09-30T00:53:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 20 client.4930.objecter put_session s=0x7fe96806e220 osd=2 4 2023-09-30T00:53:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 10 client.4930.objecter _send_command 1 2023-09-30T00:53:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 1 -- 172.21.15.40:0/1548417246 --> v2:172.21.15.40:6808/117239 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe968071350 con 0x7fe96806e8d0 2023-09-30T00:53:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.805+0000 7fe9797fa700 20 client.4930.objecter dump_active .. 0 homeless 2023-09-30T00:53:11.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.806+0000 7fe988fd7700 1 --2- 172.21.15.40:0/1548417246 >> v2:172.21.15.40:6808/117239 conn(0x7fe96806e8d0 0x7fe968070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:11.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.806+0000 7fe988fd7700 10 monclient: get_auth_request con 0x7fe96806e8d0 auth_method 0 2023-09-30T00:53:11.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.807+0000 7fe9797fa700 1 -- 172.21.15.40:0/1548417246 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe97c009240 con 0x7fe9840a7fc0 2023-09-30T00:53:11.810 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.807+0000 7fe9797fa700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:53:11.810 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.810+0000 7fe988fd7700 1 --2- 172.21.15.40:0/1548417246 >> v2:172.21.15.40:6808/117239 conn(0x7fe96806e8d0 0x7fe968070ce0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:11.810 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.810+0000 7fe9797fa700 10 client.4930.objecter ms_handle_connect 0x7fe96806e8d0 2023-09-30T00:53:11.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.837+0000 7fe9797fa700 1 -- 172.21.15.40:0/1548417246 <== osd.2 v2:172.21.15.40:6808/117239 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fe968071350 con 0x7fe96806e8d0 2023-09-30T00:53:11.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.837+0000 7fe9797fa700 10 client.4930.objecter ms_dispatch 0x7fe984001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:53:11.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.837+0000 7fe9797fa700 10 client.4930.objecter _finish_command 1 = system:0 2023-09-30T00:53:11.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.837+0000 7fe9797fa700 20 client.4930.objecter put_session s=0x7fe96806e220 osd=2 4 2023-09-30T00:53:11.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.837+0000 7fe9797fa700 15 client.4930.objecter _session_command_op_remove 2 1 2023-09-30T00:53:11.852 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.852+0000 7fe98ba3c700 10 client.4930.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T00:53:11.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.852+0000 7fe98ba3c700 15 client.4930.objecter _session_command_op_assign -1 2 2023-09-30T00:53:11.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.852+0000 7fe98ba3c700 20 client.4930.objecter _get_session s=0x7fe96806e220 osd=2 3 2023-09-30T00:53:11.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.852+0000 7fe98ba3c700 20 client.4930.objecter put_session s=0x7fe96806e220 osd=2 3 2023-09-30T00:53:11.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.852+0000 7fe98ba3c700 20 client.4930.objecter _get_session s=0x7fe96806e220 osd=2 3 2023-09-30T00:53:11.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.852+0000 7fe98ba3c700 15 client.4930.objecter _session_command_op_remove -1 2 2023-09-30T00:53:11.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.852+0000 7fe98ba3c700 20 client.4930.objecter get_session s=0x7fe96806e220 osd=2 3 2023-09-30T00:53:11.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.852+0000 7fe98ba3c700 15 client.4930.objecter _session_command_op_assign 2 2 2023-09-30T00:53:11.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.852+0000 7fe98ba3c700 20 client.4930.objecter put_session s=0x7fe96806e220 osd=2 4 2023-09-30T00:53:11.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.852+0000 7fe98ba3c700 10 client.4930.objecter _send_command 2 2023-09-30T00:53:11.853 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.852+0000 7fe98ba3c700 1 -- 172.21.15.40:0/1548417246 --> v2:172.21.15.40:6808/117239 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe9840a8c60 con 0x7fe96806e8d0 2023-09-30T00:53:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe9797fa700 1 -- 172.21.15.40:0/1548417246 <== osd.2 v2:172.21.15.40:6808/117239 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe9840a8c60 con 0x7fe96806e8d0 2023-09-30T00:53:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe9797fa700 10 client.4930.objecter ms_dispatch 0x7fe984001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:53:11.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe9797fa700 10 client.4930.objecter _finish_command 2 = system:0 2023-09-30T00:53:11.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe9797fa700 20 client.4930.objecter put_session s=0x7fe96806e220 osd=2 4 2023-09-30T00:53:11.877 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe9797fa700 15 client.4930.objecter _session_command_op_remove 2 2 2023-09-30T00:53:11.877 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-09-30T00:53:11.877 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:53:11.877 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-09-30T00:53:11.878 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe98ba3c700 10 librados: watch_flush enter 2023-09-30T00:53:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe98ba3c700 10 librados: watch_flush exit 2023-09-30T00:53:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe98ba3c700 10 client.4930.objecter close_session for osd.2 2023-09-30T00:53:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe98ba3c700 1 -- 172.21.15.40:0/1548417246 >> v2:172.21.15.40:6808/117239 conn(0x7fe96806e8d0 msgr2=0x7fe968070ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe98ba3c700 1 --2- 172.21.15.40:0/1548417246 >> v2:172.21.15.40:6808/117239 conn(0x7fe96806e8d0 0x7fe968070ce0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe98ba3c700 20 client.4930.objecter put_session s=0x7fe96806e220 osd=2 1 2023-09-30T00:53:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe98ba3c700 20 client.4930.objecter shutdown clearing up homeless session... 2023-09-30T00:53:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe98ba3c700 10 client.4930.objecter successfully canceled tick 2023-09-30T00:53:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe98ba3c700 1 -- 172.21.15.40:0/1548417246 >> v2:172.21.15.135:6800/116426 conn(0x7fe968068960 msgr2=0x7fe96806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe98ba3c700 1 --2- 172.21.15.40:0/1548417246 >> v2:172.21.15.135:6800/116426 conn(0x7fe968068960 0x7fe96806ae10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe980007fd0 tx=0x7fe9800026a0 comp rx=0 tx=0).stop 2023-09-30T00:53:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe98ba3c700 10 monclient: shutdown 2023-09-30T00:53:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe98ba3c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:11.879 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe98ba3c700 1 -- 172.21.15.40:0/1548417246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9840a7fc0 msgr2=0x7fe984140af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe98ba3c700 1 --2- 172.21.15.40:0/1548417246 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fe9840a7fc0 0x7fe984140af0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe97c00b5e0 tx=0x7fe97c009120 comp rx=0 tx=0).stop 2023-09-30T00:53:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.876+0000 7fe989fd9700 1 -- 172.21.15.40:0/1548417246 reap_dead start 2023-09-30T00:53:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.877+0000 7fe98ba3c700 1 -- 172.21.15.40:0/1548417246 shutdown_connections 2023-09-30T00:53:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.877+0000 7fe98ba3c700 1 -- 172.21.15.40:0/1548417246 >> 172.21.15.40:0/1548417246 conn(0x7fe984069060 msgr2=0x7fe9840b5140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.877+0000 7fe98ba3c700 1 -- 172.21.15.40:0/1548417246 shutdown_connections 2023-09-30T00:53:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.877+0000 7fe98ba3c700 1 -- 172.21.15.40:0/1548417246 wait complete. 2023-09-30T00:53:11.880 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:11.877+0000 7fe98ba3c700 1 librados: shutdown 2023-09-30T00:53:11.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-30T00:53:12.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.028+0000 7f5ed0b72700 10 monclient: get_monmap_and_config 2023-09-30T00:53:12.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.028+0000 7f5ed0b72700 10 monclient: build_initial_monmap 2023-09-30T00:53:12.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.028+0000 7f5ed0b72700 10 monclient: monmap: 2023-09-30T00:53:12.029 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:12.029 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:12.029 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:12.029503+0000 2023-09-30T00:53:12.029 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:12.029503+0000 2023-09-30T00:53:12.029 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:12.029 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:12.029 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:12.030 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:12.030 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:12.030 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:12.030 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:12.030 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:12.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.029+0000 7f5ed0b72700 1 Processor -- start 2023-09-30T00:53:12.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.029+0000 7f5ed0b72700 1 -- start start 2023-09-30T00:53:12.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.029+0000 7f5ed0b72700 10 monclient: init 2023-09-30T00:53:12.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.030+0000 7f5ed0b72700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:12.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.030+0000 7f5ed0b72700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-09-30T00:53:12.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.030+0000 7f5ed0b72700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ecc0b1460 addr v1:172.21.15.40:6789/0 2023-09-30T00:53:12.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.030+0000 7f5ed0b72700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ecc0b1830 0x7f5ecc0a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:12.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.030+0000 7f5ed0b72700 10 monclient(hunting): picked mon.noname-b con 0x7f5ecc0b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:12.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.030+0000 7f5ed0b72700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ecc13ac20 0x7f5ecc13b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:12.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.030+0000 7f5ed0b72700 10 monclient(hunting): picked mon.noname-a con 0x7f5ecc13ac20 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:12.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.030+0000 7f5ed0b72700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc0a5b20 con 0x7f5ecc0b1460 2023-09-30T00:53:12.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.030+0000 7f5ed0b72700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:12.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.030+0000 7f5ed0b72700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ecc141e40 con 0x7f5ecc13ac20 2023-09-30T00:53:12.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.030+0000 7f5ed0b72700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:12.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.030+0000 7f5ed0b72700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0a64f0 con 0x7f5ecc0b1830 2023-09-30T00:53:12.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.030+0000 7f5ed0b72700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:12.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.030+0000 7f5ed0b72700 10 monclient(hunting): authenticate will time out at 5653.331705s 2023-09-30T00:53:12.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.031+0000 7f5ecad9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ecc13ac20 0x7f5ecc13b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:12.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.031+0000 7f5ec9d9b700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ecc0b1830 0x7f5ecc0a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:12.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.032+0000 7f5ecad9d700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ecc13ac20 0x7f5ecc13b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46052/0 (socket says 172.21.15.40:46052) 2023-09-30T00:53:12.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.032+0000 7f5ecad9d700 1 -- 172.21.15.40:0/3622252148 learned_addr learned my addr 172.21.15.40:0/3622252148 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:12.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.032+0000 7f5ecad9d700 10 monclient(hunting): get_auth_request con 0x7f5ecc13ac20 auth_method 0 2023-09-30T00:53:12.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.032+0000 7f5ecad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:12.033 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.032+0000 7f5ecad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:12.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.032+0000 7f5ecad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:12.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.032+0000 7f5ec9d9b700 10 monclient(hunting): get_auth_request con 0x7f5ecc0b1830 auth_method 0 2023-09-30T00:53:12.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.032+0000 7f5ec9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:12.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.032+0000 7f5ec9d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:12.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.032+0000 7f5ec9d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:12.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.033+0000 7f5ec9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:12.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.033+0000 7f5ec9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:12.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.033+0000 7f5ec9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:12.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.033+0000 7f5ecad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:12.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.033+0000 7f5ecad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:12.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.033+0000 7f5ecad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:12.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.035+0000 7f5ec9d9b700 10 monclient(hunting): handle_auth_done global_id 4718 payload 274 2023-09-30T00:53:12.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.035+0000 7f5ec9d9b700 1 -- 172.21.15.40:0/3622252148 >> v2:172.21.15.40:3300/0 conn(0x7f5ecc13ac20 msgr2=0x7f5ecc13b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:12.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.035+0000 7f5ec9d9b700 1 --2- 172.21.15.40:0/3622252148 >> v2:172.21.15.40:3300/0 conn(0x7f5ecc13ac20 0x7f5ecc13b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.035+0000 7f5ec9d9b700 1 -- 172.21.15.40:0/3622252148 >> v1:172.21.15.40:6789/0 conn(0x7f5ecc0b1460 legacy=0x7f5ecc141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:12.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.035+0000 7f5ec9d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:53:12.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.035+0000 7f5ec9d9b700 1 monclient: found mon.noname-b 2023-09-30T00:53:12.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.035+0000 7f5ec9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:12.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.035+0000 7f5ec9d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:53:12.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.035+0000 7f5ec9d9b700 1 -- 172.21.15.40:0/3622252148 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc0a71f0 con 0x7f5ecc0b1830 2023-09-30T00:53:12.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.037+0000 7f5ec9d9b700 1 --2- 172.21.15.40:0/3622252148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ecc0b1830 0x7f5ecc0a5fb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5ec00098e0 tx=0x7f5ec00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb1873c61852d7fc server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:12.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ec959a700 1 -- 172.21.15.40:0/3622252148 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec000b640 con 0x7f5ecc0b1830 2023-09-30T00:53:12.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ec959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:12.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ec959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:12.043 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ec959a700 10 monclient: dump: 2023-09-30T00:53:12.043 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:12.044 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:12.044 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.044 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.044 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:12.044 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:12.044 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:12.044 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:12.044 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:12.044 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:12.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ec959a700 10 monclient: _finish_auth 0 2023-09-30T00:53:12.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ec959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:12.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ec959a700 1 -- 172.21.15.40:0/3622252148 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec000b7a0 con 0x7f5ecc0b1830 2023-09-30T00:53:12.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ed0b72700 5 monclient: authenticate success, global_id 4718 2023-09-30T00:53:12.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ed0b72700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:12.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ec959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:12.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ed0b72700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:12.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ec959a700 1 -- 172.21.15.40:0/3622252148 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec000bb40 con 0x7f5ecc0b1830 2023-09-30T00:53:12.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ed0b72700 4 set_mon_vals no callback set 2023-09-30T00:53:12.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ec959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:12.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ec959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:53:12.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.042+0000 7f5ec959a700 10 monclient: dump: 2023-09-30T00:53:12.045 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:12.046 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:12.046 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.046 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.046 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:12.046 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:12.047 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:12.047 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:12.047 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:12.047 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:12.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.043+0000 7f5ed0b72700 10 monclient: shutdown 2023-09-30T00:53:12.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.043+0000 7f5ed0b72700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:12.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.043+0000 7f5ed0b72700 1 -- 172.21.15.40:0/3622252148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ecc0b1830 msgr2=0x7f5ecc0a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:12.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.043+0000 7f5ed0b72700 1 --2- 172.21.15.40:0/3622252148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ecc0b1830 0x7f5ecc0a5fb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5ec00098e0 tx=0x7f5ec00091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.050 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.043+0000 7f5ed0b72700 1 -- 172.21.15.40:0/3622252148 shutdown_connections 2023-09-30T00:53:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.043+0000 7f5ed0b72700 1 --2- 172.21.15.40:0/3622252148 >> v2:172.21.15.40:3300/0 conn(0x7f5ecc13ac20 0x7f5ecc13b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.043+0000 7f5ed0b72700 1 --2- 172.21.15.40:0/3622252148 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ecc0b1830 0x7f5ecc0a5fb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.043+0000 7f5ed0b72700 1 -- 172.21.15.40:0/3622252148 >> 172.21.15.40:0/3622252148 conn(0x7f5ecc0631f0 msgr2=0x7f5ecc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.044+0000 7f5ed0b72700 1 -- 172.21.15.40:0/3622252148 shutdown_connections 2023-09-30T00:53:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.044+0000 7f5ed0b72700 1 -- 172.21.15.40:0/3622252148 wait complete. 2023-09-30T00:53:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.045+0000 7f5ed0b72700 10 monclient: build_initial_monmap 2023-09-30T00:53:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.045+0000 7f5ed0b72700 10 monclient: monmap: 2023-09-30T00:53:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:12.052 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.046+0000 7f5ed0b72700 1 librados: starting msgr at 2023-09-30T00:53:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.046+0000 7f5ed0b72700 1 librados: starting objecter 2023-09-30T00:53:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.046+0000 7f5ed0b72700 1 Processor -- start 2023-09-30T00:53:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.046+0000 7f5ed0b72700 1 -- start start 2023-09-30T00:53:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.046+0000 7f5ed0b72700 1 librados: setting wanted keys 2023-09-30T00:53:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.046+0000 7f5ed0b72700 1 librados: calling monclient init 2023-09-30T00:53:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.046+0000 7f5ed0b72700 10 monclient: init 2023-09-30T00:53:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:53:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ecc0b1460 0x7f5ecc13e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 10 monclient(hunting): picked mon.noname-a con 0x7f5ecc0b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ecc0b1830 0x7f5ecc0a7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 10 monclient(hunting): picked mon.noname-b con 0x7f5ecc0b1830 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ecc13ac20 0x7f5ecc150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 10 monclient(hunting): picked mon.noname-c con 0x7f5ecc13ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:12.053 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0a8680 con 0x7f5ecc0b1460 2023-09-30T00:53:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0a83a0 con 0x7f5ecc0b1830 2023-09-30T00:53:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc0ab930 con 0x7f5ecc13ac20 2023-09-30T00:53:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ed0b72700 10 monclient(hunting): authenticate will time out at 5653.348227s 2023-09-30T00:53:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5ecad9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ecc13ac20 0x7f5ecc150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.047+0000 7f5eca59c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ecc0b1460 0x7f5ecc13e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5ecad9d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ecc13ac20 0x7f5ecc150460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:37492/0 (socket says 172.21.15.40:37492) 2023-09-30T00:53:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5ecad9d700 1 -- 172.21.15.40:0/2253854184 learned_addr learned my addr 172.21.15.40:0/2253854184 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5ecad9d700 10 monclient(hunting): get_auth_request con 0x7f5ecc13ac20 auth_method 0 2023-09-30T00:53:12.054 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5ecad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:12.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5ecad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:12.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5ecad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:12.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5eca59c700 10 monclient(hunting): get_auth_request con 0x7f5ecc0b1460 auth_method 0 2023-09-30T00:53:12.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5eca59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:12.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5eca59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:12.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5eca59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:12.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5ec9d9b700 1 --2- 172.21.15.40:0/2253854184 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ecc0b1830 0x7f5ecc0a7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:12.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5ec9d9b700 10 monclient(hunting): get_auth_request con 0x7f5ecc0b1830 auth_method 0 2023-09-30T00:53:12.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5ec9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:12.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5ec9d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:12.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5ec9d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:12.055 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5ecad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:12.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5ecad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:12.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5ecad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:12.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5eca59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:12.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5eca59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:12.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.048+0000 7f5eca59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:12.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.049+0000 7f5ec9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:12.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.049+0000 7f5ec9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:12.056 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.049+0000 7f5ec9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:12.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.052+0000 7f5ec9d9b700 10 monclient(hunting): handle_auth_done global_id 4933 payload 824 2023-09-30T00:53:12.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.052+0000 7f5ec9d9b700 1 -- 172.21.15.40:0/2253854184 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ecc13ac20 msgr2=0x7f5ecc150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:12.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.052+0000 7f5ec9d9b700 1 --2- 172.21.15.40:0/2253854184 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ecc13ac20 0x7f5ecc150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.052+0000 7f5ec9d9b700 1 -- 172.21.15.40:0/2253854184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ecc0b1460 msgr2=0x7f5ecc13e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:12.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.052+0000 7f5ec9d9b700 1 --2- 172.21.15.40:0/2253854184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ecc0b1460 0x7f5ecc13e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.057 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.052+0000 7f5ec9d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:53:12.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.052+0000 7f5ec9d9b700 1 monclient: found mon.noname-b 2023-09-30T00:53:12.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.052+0000 7f5ec9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:12.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.052+0000 7f5ec9d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:12.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.052+0000 7f5ec9d9b700 1 -- 172.21.15.40:0/2253854184 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec0009420 con 0x7f5ecc0b1830 2023-09-30T00:53:12.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.054+0000 7f5ec9d9b700 1 --2- 172.21.15.40:0/2253854184 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ecc0b1830 0x7f5ecc0a7e60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5ec00098e0 tx=0x7f5ec0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:12.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.054+0000 7f5eba7fc700 10 client.?.objecter ms_handle_connect 0x7f5ecc0b1830 2023-09-30T00:53:12.058 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.054+0000 7f5eba7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:12.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.058+0000 7f5eba7fc700 1 -- 172.21.15.40:0/2253854184 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec000b910 con 0x7f5ecc0b1830 2023-09-30T00:53:12.059 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.058+0000 7f5eba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:12.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.058+0000 7f5eba7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:12.060 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.058+0000 7f5eba7fc700 10 monclient: dump: 2023-09-30T00:53:12.060 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:12.060 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:12.060 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.060 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.060 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:12.060 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:12.060 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:12.060 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:12.060 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:12.061 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:12.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.058+0000 7f5eba7fc700 10 monclient: _finish_auth 0 2023-09-30T00:53:12.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.058+0000 7f5eba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:12.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.058+0000 7f5eba7fc700 1 -- 172.21.15.40:0/2253854184 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec000ba70 con 0x7f5ecc0b1830 2023-09-30T00:53:12.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5ed0b72700 5 monclient: authenticate success, global_id 4933 2023-09-30T00:53:12.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5ed0b72700 10 monclient: _renew_subs 2023-09-30T00:53:12.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5ed0b72700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:12.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5ed0b72700 1 -- 172.21.15.40:0/2253854184 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc13eec0 con 0x7f5ecc0b1830 2023-09-30T00:53:12.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5eba7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:12.061 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5eba7fc700 1 -- 172.21.15.40:0/2253854184 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0011650 con 0x7f5ecc0b1830 2023-09-30T00:53:12.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5eba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:12.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5eba7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:12.062 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5eba7fc700 10 monclient: dump: 2023-09-30T00:53:12.062 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:12.062 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:12.062 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.062 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.062 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:12.062 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:12.063 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:12.063 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:12.063 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:12.063 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:12.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5ec8d99700 4 set_mon_vals no callback set 2023-09-30T00:53:12.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5ed0b72700 10 client.4933.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:12.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5ed0b72700 10 monclient: _renew_subs 2023-09-30T00:53:12.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5ed0b72700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:12.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5ed0b72700 1 -- 172.21.15.40:0/2253854184 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ecc150c30 con 0x7f5ecc0b1830 2023-09-30T00:53:12.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5ed0b72700 1 librados: init done 2023-09-30T00:53:12.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.059+0000 7f5ed0b72700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:12.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.060+0000 7f5ed0b72700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:12.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.060+0000 7f5ed0b72700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:12.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.060+0000 7f5ed0b72700 1 -- 172.21.15.40:0/2253854184 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ecc1502b0 con 0x7f5ecc0b1830 2023-09-30T00:53:12.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.079+0000 7f5eba7fc700 1 -- 172.21.15.40:0/2253854184 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5ec0011a00 con 0x7f5ecc0b1830 2023-09-30T00:53:12.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.079+0000 7f5eba7fc700 1 --2- 172.21.15.40:0/2253854184 >> v2:172.21.15.135:6800/116426 conn(0x7f5eac068960 0x7f5eac06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:12.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.080+0000 7f5eca59c700 1 --2- 172.21.15.40:0/2253854184 >> v2:172.21.15.135:6800/116426 conn(0x7f5eac068960 0x7f5eac06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:12.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.080+0000 7f5eca59c700 10 monclient: get_auth_request con 0x7f5eac068960 auth_method 0 2023-09-30T00:53:12.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.080+0000 7f5eca59c700 1 --2- 172.21.15.40:0/2253854184 >> v2:172.21.15.135:6800/116426 conn(0x7f5eac068960 0x7f5eac06ae10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc003a80 tx=0x7f5ebc007ec0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:12.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.080+0000 7f5eba7fc700 10 client.4933.objecter ms_handle_connect 0x7f5eac068960 2023-09-30T00:53:12.084 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.084+0000 7f5eba7fc700 1 -- 172.21.15.40:0/2253854184 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5ec0057380 con 0x7f5ecc0b1830 2023-09-30T00:53:12.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.084+0000 7f5eba7fc700 10 client.4933.objecter ms_dispatch 0x7f5ecc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:12.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.084+0000 7f5eba7fc700 3 client.4933.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:12.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.084+0000 7f5eba7fc700 3 client.4933.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:12.085 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.084+0000 7f5eba7fc700 20 client.4933.objecter dump_active .. 0 homeless 2023-09-30T00:53:12.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.323+0000 7f5eba7fc700 1 -- 172.21.15.40:0/2253854184 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ec0052fc0 con 0x7f5ecc0b1830 2023-09-30T00:53:12.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.323+0000 7f5eba7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:12.324 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.323+0000 7f5eba7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:12.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.533+0000 7f5ed0b72700 1 -- 172.21.15.40:0/2253854184 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ecc064a00 con 0x7f5eac068960 2023-09-30T00:53:12.534 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.534+0000 7f5eba7fc700 1 -- 172.21.15.40:0/2253854184 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5ecc064a00 con 0x7f5eac068960 2023-09-30T00:53:12.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 10 librados: watch_flush enter 2023-09-30T00:53:12.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 10 librados: watch_flush exit 2023-09-30T00:53:12.537 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 20 client.4933.objecter shutdown clearing up homeless session... 2023-09-30T00:53:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 10 client.4933.objecter successfully canceled tick 2023-09-30T00:53:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 1 -- 172.21.15.40:0/2253854184 >> v2:172.21.15.135:6800/116426 conn(0x7f5eac068960 msgr2=0x7f5eac06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 1 --2- 172.21.15.40:0/2253854184 >> v2:172.21.15.135:6800/116426 conn(0x7f5eac068960 0x7f5eac06ae10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc003a80 tx=0x7f5ebc007ec0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 10 monclient: shutdown 2023-09-30T00:53:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 1 -- 172.21.15.40:0/2253854184 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ecc0b1830 msgr2=0x7f5ecc0a7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 1 --2- 172.21.15.40:0/2253854184 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ecc0b1830 0x7f5ecc0a7e60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5ec00098e0 tx=0x7f5ec0017000 comp rx=0 tx=0).stop 2023-09-30T00:53:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 1 -- 172.21.15.40:0/2253854184 shutdown_connections 2023-09-30T00:53:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 1 --2- 172.21.15.40:0/2253854184 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f5ecc13ac20 0x7f5ecc150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 1 --2- 172.21.15.40:0/2253854184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ecc0b1460 0x7f5ecc13e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 1 --2- 172.21.15.40:0/2253854184 >> v2:172.21.15.135:6800/116426 conn(0x7f5eac068960 0x7f5eac06ae10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 1 --2- 172.21.15.40:0/2253854184 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f5ecc0b1830 0x7f5ecc0a7e60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 1 -- 172.21.15.40:0/2253854184 >> 172.21.15.40:0/2253854184 conn(0x7f5ecc0631f0 msgr2=0x7f5ecc0b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 1 -- 172.21.15.40:0/2253854184 shutdown_connections 2023-09-30T00:53:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 1 -- 172.21.15.40:0/2253854184 wait complete. 2023-09-30T00:53:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.536+0000 7f5ed0b72700 1 librados: shutdown 2023-09-30T00:53:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 2 to deep-scrub 2023-09-30T00:53:12.553 INFO:tasks.ceph:Scrubbing osd.3 2023-09-30T00:53:12.553 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:53:12.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.695+0000 7fc05055f700 10 monclient: get_monmap_and_config 2023-09-30T00:53:12.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.695+0000 7fc05055f700 10 monclient: build_initial_monmap 2023-09-30T00:53:12.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.695+0000 7fc05055f700 10 monclient: monmap: 2023-09-30T00:53:12.696 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:12.696 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:12.696 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:12.696172+0000 2023-09-30T00:53:12.696 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:12.696172+0000 2023-09-30T00:53:12.696 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:12.696 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:12.696 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:12.696 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:12.696 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:12.696 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:12.696 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:12.697 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:12.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.695+0000 7fc05055f700 1 Processor -- start 2023-09-30T00:53:12.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.695+0000 7fc05055f700 1 -- start start 2023-09-30T00:53:12.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.695+0000 7fc05055f700 10 monclient: init 2023-09-30T00:53:12.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc05055f700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:12.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc05055f700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-09-30T00:53:12.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc05055f700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc0480a5710 0x7fc0480b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:12.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc05055f700 10 monclient(hunting): picked mon.noname-a con 0x7fc0480a5710 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:12.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc05055f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc0480a7630 addr v1:172.21.15.40:6789/0 2023-09-30T00:53:12.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc05055f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc04813af10 0x7fc04813b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:12.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc05055f700 10 monclient(hunting): picked mon.noname-b con 0x7fc04813af10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:12.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc05055f700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0480b1770 con 0x7fc0480a7630 2023-09-30T00:53:12.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc05055f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:12.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc05055f700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0480a5ae0 con 0x7fc0480a5710 2023-09-30T00:53:12.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc05055f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:12.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc05055f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480b1040 con 0x7fc04813af10 2023-09-30T00:53:12.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc05055f700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:12.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc05055f700 10 monclient(hunting): authenticate will time out at 5653.998232s 2023-09-30T00:53:12.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.697+0000 7fc04eafc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc04813af10 0x7fc04813b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:12.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.698+0000 7fc04eafc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc04813af10 0x7fc04813b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:37498/0 (socket says 172.21.15.40:37498) 2023-09-30T00:53:12.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.698+0000 7fc04eafc700 1 -- 172.21.15.40:0/2167619556 learned_addr learned my addr 172.21.15.40:0/2167619556 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:12.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.698+0000 7fc04eafc700 10 monclient(hunting): get_auth_request con 0x7fc04813af10 auth_method 0 2023-09-30T00:53:12.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.698+0000 7fc04eafc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:12.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.698+0000 7fc04eafc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:12.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.698+0000 7fc04eafc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:12.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.698+0000 7fc04eafc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:12.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.698+0000 7fc04eafc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:12.702 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.698+0000 7fc04eafc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:12.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.701+0000 7fc04eafc700 10 monclient(hunting): handle_auth_done global_id 4724 payload 274 2023-09-30T00:53:12.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.701+0000 7fc04eafc700 1 -- 172.21.15.40:0/2167619556 >> v2:172.21.15.40:3300/0 conn(0x7fc0480a5710 msgr2=0x7fc0480b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:12.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.701+0000 7fc04eafc700 1 --2- 172.21.15.40:0/2167619556 >> v2:172.21.15.40:3300/0 conn(0x7fc0480a5710 0x7fc0480b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.701+0000 7fc04eafc700 1 -- 172.21.15.40:0/2167619556 >> v1:172.21.15.40:6789/0 conn(0x7fc0480a7630 legacy=0x7fc0480a7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:12.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.701+0000 7fc04eafc700 10 monclient: _finish_hunting 0 2023-09-30T00:53:12.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.701+0000 7fc04eafc700 1 monclient: found mon.noname-b 2023-09-30T00:53:12.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.701+0000 7fc04eafc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:12.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.701+0000 7fc04eafc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:53:12.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.701+0000 7fc04eafc700 1 -- 172.21.15.40:0/2167619556 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0480b1440 con 0x7fc04813af10 2023-09-30T00:53:12.704 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.704+0000 7fc04eafc700 1 --2- 172.21.15.40:0/2167619556 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc04813af10 0x7fc04813b370 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc040009990 tx=0x7fc040009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7c99f304cc09f3b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:12.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc04d2f9700 1 -- 172.21.15.40:0/2167619556 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040005c60 con 0x7fc04813af10 2023-09-30T00:53:12.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc04d2f9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:12.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc04d2f9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:12.712 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc04d2f9700 10 monclient: dump: 2023-09-30T00:53:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:12.713 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc04d2f9700 10 monclient: _finish_auth 0 2023-09-30T00:53:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc04d2f9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc04d2f9700 1 -- 172.21.15.40:0/2167619556 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc040005dc0 con 0x7fc04813af10 2023-09-30T00:53:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc05055f700 5 monclient: authenticate success, global_id 4724 2023-09-30T00:53:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc05055f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc04d2f9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc05055f700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc04d2f9700 1 -- 172.21.15.40:0/2167619556 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04000b860 con 0x7fc04813af10 2023-09-30T00:53:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc05055f700 4 set_mon_vals no callback set 2023-09-30T00:53:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc04d2f9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc04d2f9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:53:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc04d2f9700 10 monclient: dump: 2023-09-30T00:53:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.715 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.715 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:12.715 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:12.715 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:12.715 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:12.715 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:12.715 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:12.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc05055f700 10 monclient: shutdown 2023-09-30T00:53:12.718 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc05055f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:12.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc05055f700 1 -- 172.21.15.40:0/2167619556 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc04813af10 msgr2=0x7fc04813b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:12.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.712+0000 7fc05055f700 1 --2- 172.21.15.40:0/2167619556 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc04813af10 0x7fc04813b370 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc040009990 tx=0x7fc040009250 comp rx=0 tx=0).stop 2023-09-30T00:53:12.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.713+0000 7fc05055f700 1 -- 172.21.15.40:0/2167619556 shutdown_connections 2023-09-30T00:53:12.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.713+0000 7fc05055f700 1 --2- 172.21.15.40:0/2167619556 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc04813af10 0x7fc04813b370 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.713+0000 7fc05055f700 1 --2- 172.21.15.40:0/2167619556 >> v2:172.21.15.40:3300/0 conn(0x7fc0480a5710 0x7fc0480b0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.713+0000 7fc05055f700 1 -- 172.21.15.40:0/2167619556 >> 172.21.15.40:0/2167619556 conn(0x7fc048069060 msgr2=0x7fc0480631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:12.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.713+0000 7fc05055f700 1 -- 172.21.15.40:0/2167619556 shutdown_connections 2023-09-30T00:53:12.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.713+0000 7fc05055f700 1 -- 172.21.15.40:0/2167619556 wait complete. 2023-09-30T00:53:12.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.714+0000 7fc05055f700 10 monclient: build_initial_monmap 2023-09-30T00:53:12.719 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.714+0000 7fc05055f700 10 monclient: monmap: 2023-09-30T00:53:12.719 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:12.719 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:12.720 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:12.720 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:12.720 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:12.720 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:12.720 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:12.720 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:12.720 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:12.720 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:12.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.714+0000 7fc05055f700 1 librados: starting msgr at 2023-09-30T00:53:12.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.714+0000 7fc05055f700 1 librados: starting objecter 2023-09-30T00:53:12.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.714+0000 7fc05055f700 1 Processor -- start 2023-09-30T00:53:12.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.714+0000 7fc05055f700 1 -- start start 2023-09-30T00:53:12.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.714+0000 7fc05055f700 1 librados: setting wanted keys 2023-09-30T00:53:12.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.714+0000 7fc05055f700 1 librados: calling monclient init 2023-09-30T00:53:12.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.714+0000 7fc05055f700 10 monclient: init 2023-09-30T00:53:12.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:12.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:53:12.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0480a5710 0x7fc04813c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:12.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 10 monclient(hunting): picked mon.noname-b con 0x7fc0480a5710 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:12.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0480a7630 0x7fc04813ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:12.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 10 monclient(hunting): picked mon.noname-a con 0x7fc0480a7630 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:12.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc04813af10 0x7fc0480ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:12.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 10 monclient(hunting): picked mon.noname-c con 0x7fc04813af10 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:12.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:12.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480aaab0 con 0x7fc0480a7630 2023-09-30T00:53:12.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:12.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480aa7d0 con 0x7fc0480a5710 2023-09-30T00:53:12.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:12.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480a99c0 con 0x7fc04813af10 2023-09-30T00:53:12.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:12.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc05055f700 10 monclient(hunting): authenticate will time out at 5654.016390s 2023-09-30T00:53:12.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.715+0000 7fc04e2fb700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0480a5710 0x7fc04813c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:12.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.716+0000 7fc04e2fb700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0480a5710 0x7fc04813c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:56502/0 (socket says 172.21.15.40:56502) 2023-09-30T00:53:12.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.716+0000 7fc04e2fb700 1 -- 172.21.15.40:0/969764208 learned_addr learned my addr 172.21.15.40:0/969764208 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:12.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.716+0000 7fc04e2fb700 10 monclient(hunting): get_auth_request con 0x7fc0480a5710 auth_method 0 2023-09-30T00:53:12.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.716+0000 7fc04e2fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:12.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.716+0000 7fc04e2fb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:12.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.716+0000 7fc04e2fb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:12.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.716+0000 7fc04eafc700 1 --2- 172.21.15.40:0/969764208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc04813af10 0x7fc0480ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:12.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.716+0000 7fc04eafc700 10 monclient(hunting): get_auth_request con 0x7fc04813af10 auth_method 0 2023-09-30T00:53:12.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.716+0000 7fc04eafc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:12.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.716+0000 7fc04eafc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:12.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.716+0000 7fc04eafc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:12.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.716+0000 7fc04e2fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:12.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.716+0000 7fc04e2fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:12.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.716+0000 7fc04e2fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:12.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.717+0000 7fc04eafc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:12.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.717+0000 7fc04eafc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:12.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.717+0000 7fc04eafc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:12.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.720+0000 7fc04e2fb700 10 monclient(hunting): handle_auth_done global_id 4936 payload 824 2023-09-30T00:53:12.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.720+0000 7fc04e2fb700 1 -- 172.21.15.40:0/969764208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc04813af10 msgr2=0x7fc0480ab0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:12.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.720+0000 7fc04e2fb700 1 --2- 172.21.15.40:0/969764208 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc04813af10 0x7fc0480ab0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.720+0000 7fc04e2fb700 1 -- 172.21.15.40:0/969764208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0480a7630 msgr2=0x7fc04813ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.720+0000 7fc04e2fb700 1 --2- 172.21.15.40:0/969764208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0480a7630 0x7fc04813ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.720+0000 7fc04e2fb700 10 monclient: _finish_hunting 0 2023-09-30T00:53:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.720+0000 7fc04e2fb700 1 monclient: found mon.noname-b 2023-09-30T00:53:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.720+0000 7fc04e2fb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.720+0000 7fc04e2fb700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.720+0000 7fc04e2fb700 1 -- 172.21.15.40:0/969764208 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0400094d0 con 0x7fc0480a5710 2023-09-30T00:53:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.722+0000 7fc04e2fb700 1 --2- 172.21.15.40:0/969764208 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0480a5710 0x7fc04813c2b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc04400bc60 tx=0x7fc044009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.722+0000 7fc03e7fc700 10 client.?.objecter ms_handle_connect 0x7fc0480a5710 2023-09-30T00:53:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.722+0000 7fc03e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:12.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.726+0000 7fc03e7fc700 1 -- 172.21.15.40:0/969764208 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0440092e0 con 0x7fc0480a5710 2023-09-30T00:53:12.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.727+0000 7fc03e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:12.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.727+0000 7fc03e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.727+0000 7fc03e7fc700 10 monclient: dump: 2023-09-30T00:53:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:12.728 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.727+0000 7fc03e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:53:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.727+0000 7fc03e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.727+0000 7fc05055f700 5 monclient: authenticate success, global_id 4936 2023-09-30T00:53:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.727+0000 7fc05055f700 10 monclient: _renew_subs 2023-09-30T00:53:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.727+0000 7fc05055f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.727+0000 7fc05055f700 1 -- 172.21.15.40:0/969764208 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc048151090 con 0x7fc0480a5710 2023-09-30T00:53:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.727+0000 7fc05055f700 10 client.4936.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.727+0000 7fc05055f700 10 monclient: _renew_subs 2023-09-30T00:53:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.727+0000 7fc05055f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.727+0000 7fc05055f700 1 -- 172.21.15.40:0/969764208 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0481515a0 con 0x7fc0480a5710 2023-09-30T00:53:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.727+0000 7fc05055f700 1 librados: init done 2023-09-30T00:53:12.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.728+0000 7fc05055f700 10 client.4936.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:12.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.728+0000 7fc05055f700 15 client.4936.objecter _session_command_op_assign -1 1 2023-09-30T00:53:12.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.728+0000 7fc05055f700 20 client.4936.objecter _get_session osd=-1 returning homeless 2023-09-30T00:53:12.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.728+0000 7fc05055f700 10 client.4936.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:12.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.728+0000 7fc05055f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:12.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.728+0000 7fc05055f700 1 -- 172.21.15.40:0/969764208 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc0480aa5f0 con 0x7fc0480a5710 2023-09-30T00:53:12.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.815+0000 7fc03e7fc700 1 -- 172.21.15.40:0/969764208 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0480aa5f0 con 0x7fc0480a5710 2023-09-30T00:53:12.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.815+0000 7fc03e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:12.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.815+0000 7fc03e7fc700 1 -- 172.21.15.40:0/969764208 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04400c8f0 con 0x7fc0480a5710 2023-09-30T00:53:12.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.815+0000 7fc03e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:12.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.815+0000 7fc03e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:12.817 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.815+0000 7fc03e7fc700 10 monclient: dump: 2023-09-30T00:53:12.817 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:12.817 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:12.817 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.817 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:12.817 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:12.817 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:12.817 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:12.818 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:12.818 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:12.818 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:12.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.815+0000 7fc04caf8700 4 set_mon_vals no callback set 2023-09-30T00:53:12.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.816+0000 7fc03e7fc700 1 -- 172.21.15.40:0/969764208 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fc044012070 con 0x7fc0480a5710 2023-09-30T00:53:12.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.816+0000 7fc03e7fc700 1 --2- 172.21.15.40:0/969764208 >> v2:172.21.15.135:6800/116426 conn(0x7fc02c068890 0x7fc02c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:12.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.816+0000 7fc04dafa700 1 --2- 172.21.15.40:0/969764208 >> v2:172.21.15.135:6800/116426 conn(0x7fc02c068890 0x7fc02c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:12.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.816+0000 7fc04dafa700 10 monclient: get_auth_request con 0x7fc02c068890 auth_method 0 2023-09-30T00:53:12.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.817+0000 7fc04dafa700 1 --2- 172.21.15.40:0/969764208 >> v2:172.21.15.135:6800/116426 conn(0x7fc02c068890 0x7fc02c06ad40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc04813c7f0 tx=0x7fc038006ce0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:12.819 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.817+0000 7fc03e7fc700 10 client.4936.objecter ms_handle_connect 0x7fc02c068890 2023-09-30T00:53:12.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 1 -- 172.21.15.40:0/969764208 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fc0440554d0 con 0x7fc0480a5710 2023-09-30T00:53:12.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 10 client.4936.objecter ms_dispatch 0x7fc048001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:12.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 3 client.4936.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:12.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 3 client.4936.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:12.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 10 client.4936.objecter checking command 1 2023-09-30T00:53:12.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 1 --2- 172.21.15.40:0/969764208 >> v2:172.21.15.40:6812/117236 conn(0x7fc02c06e800 0x7fc02c070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:12.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 20 client.4936.objecter _get_session s=0x7fc02c06e150 osd=3 3 2023-09-30T00:53:12.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 20 client.4936.objecter put_session s=0x7fc02c06e150 osd=3 3 2023-09-30T00:53:12.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 15 client.4936.objecter _session_command_op_remove -1 1 2023-09-30T00:53:12.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 20 client.4936.objecter _get_session s=0x7fc02c06e150 osd=3 3 2023-09-30T00:53:12.825 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 20 client.4936.objecter get_session s=0x7fc02c06e150 osd=3 3 2023-09-30T00:53:12.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 15 client.4936.objecter _session_command_op_assign 3 1 2023-09-30T00:53:12.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 20 client.4936.objecter put_session s=0x7fc02c06e150 osd=3 4 2023-09-30T00:53:12.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 10 client.4936.objecter _send_command 1 2023-09-30T00:53:12.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 1 -- 172.21.15.40:0/969764208 --> v2:172.21.15.40:6812/117236 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc02c071280 con 0x7fc02c06e800 2023-09-30T00:53:12.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.823+0000 7fc03e7fc700 20 client.4936.objecter dump_active .. 0 homeless 2023-09-30T00:53:12.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.824+0000 7fc04eafc700 1 --2- 172.21.15.40:0/969764208 >> v2:172.21.15.40:6812/117236 conn(0x7fc02c06e800 0x7fc02c070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:12.826 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.825+0000 7fc04eafc700 10 monclient: get_auth_request con 0x7fc02c06e800 auth_method 0 2023-09-30T00:53:12.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.828+0000 7fc03e7fc700 10 client.4936.objecter ms_handle_connect 0x7fc02c06e800 2023-09-30T00:53:12.828 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.828+0000 7fc04eafc700 1 --2- 172.21.15.40:0/969764208 >> v2:172.21.15.40:6812/117236 conn(0x7fc02c06e800 0x7fc02c070c10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:12.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.832+0000 7fc03e7fc700 1 -- 172.21.15.40:0/969764208 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc044051110 con 0x7fc0480a5710 2023-09-30T00:53:12.833 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.832+0000 7fc03e7fc700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:53:12.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.854+0000 7fc03e7fc700 1 -- 172.21.15.40:0/969764208 <== osd.3 v2:172.21.15.40:6812/117236 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fc02c071280 con 0x7fc02c06e800 2023-09-30T00:53:12.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.854+0000 7fc03e7fc700 10 client.4936.objecter ms_dispatch 0x7fc048001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:53:12.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.854+0000 7fc03e7fc700 10 client.4936.objecter _finish_command 1 = system:0 2023-09-30T00:53:12.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.854+0000 7fc03e7fc700 20 client.4936.objecter put_session s=0x7fc02c06e150 osd=3 4 2023-09-30T00:53:12.855 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.854+0000 7fc03e7fc700 15 client.4936.objecter _session_command_op_remove 3 1 2023-09-30T00:53:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.883+0000 7fc05055f700 10 client.4936.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T00:53:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.883+0000 7fc05055f700 15 client.4936.objecter _session_command_op_assign -1 2 2023-09-30T00:53:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.883+0000 7fc05055f700 20 client.4936.objecter _get_session s=0x7fc02c06e150 osd=3 3 2023-09-30T00:53:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.883+0000 7fc05055f700 20 client.4936.objecter put_session s=0x7fc02c06e150 osd=3 3 2023-09-30T00:53:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.883+0000 7fc05055f700 20 client.4936.objecter _get_session s=0x7fc02c06e150 osd=3 3 2023-09-30T00:53:12.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.883+0000 7fc05055f700 15 client.4936.objecter _session_command_op_remove -1 2 2023-09-30T00:53:12.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.883+0000 7fc05055f700 20 client.4936.objecter get_session s=0x7fc02c06e150 osd=3 3 2023-09-30T00:53:12.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.883+0000 7fc05055f700 15 client.4936.objecter _session_command_op_assign 3 2 2023-09-30T00:53:12.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.883+0000 7fc05055f700 20 client.4936.objecter put_session s=0x7fc02c06e150 osd=3 4 2023-09-30T00:53:12.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.883+0000 7fc05055f700 10 client.4936.objecter _send_command 2 2023-09-30T00:53:12.885 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.883+0000 7fc05055f700 1 -- 172.21.15.40:0/969764208 --> v2:172.21.15.40:6812/117236 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc048064a00 con 0x7fc02c06e800 2023-09-30T00:53:12.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.910+0000 7fc03e7fc700 1 -- 172.21.15.40:0/969764208 <== osd.3 v2:172.21.15.40:6812/117236 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc048064a00 con 0x7fc02c06e800 2023-09-30T00:53:12.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.910+0000 7fc03e7fc700 10 client.4936.objecter ms_dispatch 0x7fc048001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:53:12.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.910+0000 7fc03e7fc700 10 client.4936.objecter _finish_command 2 = system:0 2023-09-30T00:53:12.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.910+0000 7fc03e7fc700 20 client.4936.objecter put_session s=0x7fc02c06e150 osd=3 4 2023-09-30T00:53:12.911 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.910+0000 7fc03e7fc700 15 client.4936.objecter _session_command_op_remove 3 2 2023-09-30T00:53:12.911 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-09-30T00:53:12.912 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:53:12.912 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-09-30T00:53:12.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 10 librados: watch_flush enter 2023-09-30T00:53:12.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 10 librados: watch_flush exit 2023-09-30T00:53:12.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 10 client.4936.objecter close_session for osd.3 2023-09-30T00:53:12.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 1 -- 172.21.15.40:0/969764208 >> v2:172.21.15.40:6812/117236 conn(0x7fc02c06e800 msgr2=0x7fc02c070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:12.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 1 --2- 172.21.15.40:0/969764208 >> v2:172.21.15.40:6812/117236 conn(0x7fc02c06e800 0x7fc02c070c10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 20 client.4936.objecter put_session s=0x7fc02c06e150 osd=3 1 2023-09-30T00:53:12.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 20 client.4936.objecter shutdown clearing up homeless session... 2023-09-30T00:53:12.913 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 10 client.4936.objecter successfully canceled tick 2023-09-30T00:53:12.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 1 -- 172.21.15.40:0/969764208 >> v2:172.21.15.135:6800/116426 conn(0x7fc02c068890 msgr2=0x7fc02c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:12.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 1 --2- 172.21.15.40:0/969764208 >> v2:172.21.15.135:6800/116426 conn(0x7fc02c068890 0x7fc02c06ad40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc04813c7f0 tx=0x7fc038006ce0 comp rx=0 tx=0).stop 2023-09-30T00:53:12.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 10 monclient: shutdown 2023-09-30T00:53:12.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:12.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 1 -- 172.21.15.40:0/969764208 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0480a5710 msgr2=0x7fc04813c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:12.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 1 --2- 172.21.15.40:0/969764208 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc0480a5710 0x7fc04813c2b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc04400bc60 tx=0x7fc044009840 comp rx=0 tx=0).stop 2023-09-30T00:53:12.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc04eafc700 1 -- 172.21.15.40:0/969764208 reap_dead start 2023-09-30T00:53:12.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 1 -- 172.21.15.40:0/969764208 shutdown_connections 2023-09-30T00:53:12.914 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 1 -- 172.21.15.40:0/969764208 >> 172.21.15.40:0/969764208 conn(0x7fc048069060 msgr2=0x7fc0480af2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:12.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 1 -- 172.21.15.40:0/969764208 shutdown_connections 2023-09-30T00:53:12.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 1 -- 172.21.15.40:0/969764208 wait complete. 2023-09-30T00:53:12.915 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:12.911+0000 7fc05055f700 1 librados: shutdown 2023-09-30T00:53:12.926 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-09-30T00:53:13.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.063+0000 7fdf2763d700 10 monclient: get_monmap_and_config 2023-09-30T00:53:13.063 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.063+0000 7fdf2763d700 10 monclient: build_initial_monmap 2023-09-30T00:53:13.064 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.063+0000 7fdf2763d700 10 monclient: monmap: 2023-09-30T00:53:13.064 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:13.064 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:13.064 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:13.064214+0000 2023-09-30T00:53:13.064 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:13.064214+0000 2023-09-30T00:53:13.064 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:13.064 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:13.064 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:13.064 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:13.065 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:13.065 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:13.065 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:13.065 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:13.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.063+0000 7fdf2763d700 1 Processor -- start 2023-09-30T00:53:13.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.064+0000 7fdf2763d700 1 -- start start 2023-09-30T00:53:13.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.064+0000 7fdf2763d700 10 monclient: init 2023-09-30T00:53:13.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.064+0000 7fdf2763d700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:13.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.064+0000 7fdf2763d700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-09-30T00:53:13.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.064+0000 7fdf2763d700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fdf200b1460 0x7fdf200b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:13.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.064+0000 7fdf2763d700 10 monclient(hunting): picked mon.noname-c con 0x7fdf200b1460 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:13.067 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.064+0000 7fdf2763d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf200a5fb0 addr v1:172.21.15.40:6789/0 2023-09-30T00:53:13.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.064+0000 7fdf2763d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf2013ac20 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:13.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.064+0000 7fdf2763d700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf200a5b20 con 0x7fdf200a5fb0 2023-09-30T00:53:13.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.064+0000 7fdf2763d700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf200a71f0 con 0x7fdf2013ac20 2023-09-30T00:53:13.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.064+0000 7fdf2763d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:13.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.064+0000 7fdf2763d700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf200a74d0 con 0x7fdf200b1460 2023-09-30T00:53:13.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.064+0000 7fdf2763d700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:13.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.064+0000 7fdf2763d700 10 monclient(hunting): authenticate will time out at 5654.365818s 2023-09-30T00:53:13.068 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.065+0000 7fdf24bd8700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdf200a5fb0 0x7fdf20141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53290/0 (socket says 172.21.15.40:53290) 2023-09-30T00:53:13.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.065+0000 7fdf24bd8700 1 -- 172.21.15.40:0/2192431384 learned_addr learned my addr 172.21.15.40:0/2192431384 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:53:13.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.065+0000 7fdf253d9700 1 --2- 172.21.15.40:0/2192431384 >> v2:172.21.15.135:3300/0 conn(0x7fdf200b1460 0x7fdf200b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:13.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.065+0000 7fdf253d9700 10 monclient(hunting): get_auth_request con 0x7fdf200b1460 auth_method 0 2023-09-30T00:53:13.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.065+0000 7fdf253d9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:13.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.065+0000 7fdf253d9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:13.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.065+0000 7fdf253d9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:13.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.066+0000 7fdf253d9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:13.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.066+0000 7fdf253d9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:13.069 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.066+0000 7fdf253d9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:13.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.068+0000 7fdf253d9700 10 monclient(hunting): handle_auth_done global_id 4939 payload 274 2023-09-30T00:53:13.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.068+0000 7fdf253d9700 1 -- 172.21.15.40:0/2192431384 >> v1:172.21.15.135:6789/0 conn(0x7fdf2013ac20 legacy=0x7fdf2013b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:13.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.068+0000 7fdf253d9700 1 -- 172.21.15.40:0/2192431384 >> v1:172.21.15.40:6789/0 conn(0x7fdf200a5fb0 legacy=0x7fdf20141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:13.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.068+0000 7fdf253d9700 10 monclient: _finish_hunting 0 2023-09-30T00:53:13.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.068+0000 7fdf253d9700 1 monclient: found mon.noname-c 2023-09-30T00:53:13.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.068+0000 7fdf253d9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:13.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.068+0000 7fdf253d9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:53:13.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.068+0000 7fdf253d9700 1 -- 172.21.15.40:0/2192431384 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf200a63c0 con 0x7fdf200b1460 2023-09-30T00:53:13.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.068+0000 7fdf17fff700 1 -- 172.21.15.40:0/2192431384 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619422453 0 0) 0x7fdf200a5b20 con 0x7fdf200a5fb0 2023-09-30T00:53:13.071 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.068+0000 7fdf17fff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T00:53:13.072 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.072+0000 7fdf253d9700 1 --2- 172.21.15.40:0/2192431384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf200b1460 0x7fdf200b1830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c009910 tx=0x7fdf1c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=593a80283a7f2755 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf17fff700 1 -- 172.21.15.40:0/2192431384 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c00b7e0 con 0x7fdf200b1460 2023-09-30T00:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf17fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf17fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf17fff700 10 monclient: dump: 2023-09-30T00:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.079 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf17fff700 10 monclient: _finish_auth 0 2023-09-30T00:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf17fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf17fff700 1 -- 172.21.15.40:0/2192431384 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1c00b940 con 0x7fdf200b1460 2023-09-30T00:53:13.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf2763d700 5 monclient: authenticate success, global_id 4939 2023-09-30T00:53:13.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf2763d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:13.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf17fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:13.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf2763d700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:13.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf2763d700 4 set_mon_vals no callback set 2023-09-30T00:53:13.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf17fff700 1 -- 172.21.15.40:0/2192431384 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c00bce0 con 0x7fdf200b1460 2023-09-30T00:53:13.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf17fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:13.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf17fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:13.081 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.078+0000 7fdf17fff700 10 monclient: dump: 2023-09-30T00:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:13.082 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:13.083 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.079+0000 7fdf2763d700 10 monclient: shutdown 2023-09-30T00:53:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.079+0000 7fdf2763d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.079+0000 7fdf2763d700 1 -- 172.21.15.40:0/2192431384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf200b1460 msgr2=0x7fdf200b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.079+0000 7fdf2763d700 1 --2- 172.21.15.40:0/2192431384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf200b1460 0x7fdf200b1830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c009910 tx=0x7fdf1c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.079+0000 7fdf2763d700 1 -- 172.21.15.40:0/2192431384 shutdown_connections 2023-09-30T00:53:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.079+0000 7fdf2763d700 1 --2- 172.21.15.40:0/2192431384 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf200b1460 0x7fdf200b1830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.079+0000 7fdf2763d700 1 -- 172.21.15.40:0/2192431384 >> 172.21.15.40:0/2192431384 conn(0x7fdf200631f0 msgr2=0x7fdf200635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.079+0000 7fdf2763d700 1 -- 172.21.15.40:0/2192431384 shutdown_connections 2023-09-30T00:53:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.079+0000 7fdf2763d700 1 -- 172.21.15.40:0/2192431384 wait complete. 2023-09-30T00:53:13.086 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.080+0000 7fdf2763d700 10 monclient: build_initial_monmap 2023-09-30T00:53:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.080+0000 7fdf2763d700 10 monclient: monmap: 2023-09-30T00:53:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:13.087 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:13.088 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:13.088 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:13.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.080+0000 7fdf2763d700 1 librados: starting msgr at 2023-09-30T00:53:13.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.080+0000 7fdf2763d700 1 librados: starting objecter 2023-09-30T00:53:13.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 1 Processor -- start 2023-09-30T00:53:13.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 1 -- start start 2023-09-30T00:53:13.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 1 librados: setting wanted keys 2023-09-30T00:53:13.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 1 librados: calling monclient init 2023-09-30T00:53:13.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 10 monclient: init 2023-09-30T00:53:13.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:13.088 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:53:13.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf200a5fb0 0x7fdf2013dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:13.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 10 monclient(hunting): picked mon.noname-b con 0x7fdf200a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:13.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf200b1460 0x7fdf200a9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:13.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 10 monclient(hunting): picked mon.noname-a con 0x7fdf200b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:13.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf2013ac20 0x7fdf2014d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:13.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 10 monclient(hunting): picked mon.noname-c con 0x7fdf2013ac20 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:13.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:13.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf200a9940 con 0x7fdf200b1460 2023-09-30T00:53:13.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:13.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf200a9c10 con 0x7fdf200a5fb0 2023-09-30T00:53:13.089 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:13.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf200ab380 con 0x7fdf2013ac20 2023-09-30T00:53:13.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:13.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.081+0000 7fdf2763d700 10 monclient(hunting): authenticate will time out at 5654.382901s 2023-09-30T00:53:13.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.082+0000 7fdf25bda700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf2013ac20 0x7fdf2014d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:13.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.082+0000 7fdf24bd8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf200b1460 0x7fdf200a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:13.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.082+0000 7fdf25bda700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf2013ac20 0x7fdf2014d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:37528/0 (socket says 172.21.15.40:37528) 2023-09-30T00:53:13.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.082+0000 7fdf25bda700 1 -- 172.21.15.40:0/4217105522 learned_addr learned my addr 172.21.15.40:0/4217105522 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:13.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.082+0000 7fdf25bda700 10 monclient(hunting): get_auth_request con 0x7fdf2013ac20 auth_method 0 2023-09-30T00:53:13.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.082+0000 7fdf25bda700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:13.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.082+0000 7fdf25bda700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:13.090 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.082+0000 7fdf25bda700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:13.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.083+0000 7fdf24bd8700 10 monclient(hunting): get_auth_request con 0x7fdf200b1460 auth_method 0 2023-09-30T00:53:13.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.083+0000 7fdf24bd8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:13.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.083+0000 7fdf24bd8700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:13.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.083+0000 7fdf24bd8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:13.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.083+0000 7fdf25bda700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:13.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.083+0000 7fdf25bda700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:13.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.083+0000 7fdf25bda700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:13.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.083+0000 7fdf253d9700 1 --2- 172.21.15.40:0/4217105522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf200a5fb0 0x7fdf2013dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:13.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.084+0000 7fdf24bd8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:13.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.084+0000 7fdf24bd8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:13.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.084+0000 7fdf24bd8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:13.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.085+0000 7fdf253d9700 10 monclient(hunting): get_auth_request con 0x7fdf200a5fb0 auth_method 0 2023-09-30T00:53:13.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.085+0000 7fdf253d9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:13.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.085+0000 7fdf253d9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:13.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.085+0000 7fdf253d9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:13.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.085+0000 7fdf253d9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:13.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.085+0000 7fdf253d9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:13.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.085+0000 7fdf253d9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:13.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.087+0000 7fdf25bda700 10 monclient(hunting): handle_auth_done global_id 4730 payload 824 2023-09-30T00:53:13.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.087+0000 7fdf25bda700 1 -- 172.21.15.40:0/4217105522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf200a5fb0 msgr2=0x7fdf2013dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:13.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.087+0000 7fdf25bda700 1 --2- 172.21.15.40:0/4217105522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf200a5fb0 0x7fdf2013dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.087+0000 7fdf25bda700 1 -- 172.21.15.40:0/4217105522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf200b1460 msgr2=0x7fdf200a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:13.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.087+0000 7fdf25bda700 1 --2- 172.21.15.40:0/4217105522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf200b1460 0x7fdf200a9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.087+0000 7fdf25bda700 10 monclient: _finish_hunting 0 2023-09-30T00:53:13.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.087+0000 7fdf25bda700 1 monclient: found mon.noname-c 2023-09-30T00:53:13.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.087+0000 7fdf25bda700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:13.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.087+0000 7fdf25bda700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.40:3301/0 2023-09-30T00:53:13.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.087+0000 7fdf25bda700 1 -- 172.21.15.40:0/4217105522 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c009420 con 0x7fdf2013ac20 2023-09-30T00:53:13.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.090+0000 7fdf25bda700 1 --2- 172.21.15.40:0/4217105522 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf2013ac20 0x7fdf2014d8f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800bc30 tx=0x7fdf18009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:13.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.090+0000 7fdf157fa700 10 client.?.objecter ms_handle_connect 0x7fdf2013ac20 2023-09-30T00:53:13.094 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.090+0000 7fdf157fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:13.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.096+0000 7fdf157fa700 1 -- 172.21.15.40:0/4217105522 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1800c7e0 con 0x7fdf2013ac20 2023-09-30T00:53:13.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.096+0000 7fdf157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:13.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.096+0000 7fdf157fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:53:13.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.096+0000 7fdf157fa700 10 monclient: dump: 2023-09-30T00:53:13.098 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:13.098 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:13.098 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.098 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.098 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:13.098 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:13.098 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:13.098 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:13.099 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:13.099 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:13.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.096+0000 7fdf157fa700 10 monclient: _finish_auth 0 2023-09-30T00:53:13.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.096+0000 7fdf157fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:13.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.096+0000 7fdf157fa700 1 -- 172.21.15.40:0/4217105522 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1800c980 con 0x7fdf2013ac20 2023-09-30T00:53:13.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.096+0000 7fdf2763d700 5 monclient: authenticate success, global_id 4730 2023-09-30T00:53:13.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.096+0000 7fdf2763d700 10 monclient: _renew_subs 2023-09-30T00:53:13.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.096+0000 7fdf2763d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:53:13.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.096+0000 7fdf2763d700 1 -- 172.21.15.40:0/4217105522 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2013e960 con 0x7fdf2013ac20 2023-09-30T00:53:13.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.096+0000 7fdf157fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:13.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.096+0000 7fdf157fa700 1 -- 172.21.15.40:0/4217105522 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1800cd40 con 0x7fdf2013ac20 2023-09-30T00:53:13.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.096+0000 7fdf157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:13.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.097+0000 7fdf157fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:53:13.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.097+0000 7fdf157fa700 10 monclient: dump: 2023-09-30T00:53:13.100 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:13.100 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:13.100 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.100 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.100 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:13.100 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:13.100 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:13.100 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:13.100 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:13.100 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:13.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.097+0000 7fdf2763d700 10 client.4730.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:13.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.097+0000 7fdf2763d700 10 monclient: _renew_subs 2023-09-30T00:53:13.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.097+0000 7fdf2763d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:53:13.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.097+0000 7fdf2763d700 1 -- 172.21.15.40:0/4217105522 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2014de30 con 0x7fdf2013ac20 2023-09-30T00:53:13.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.097+0000 7fdf177fe700 4 set_mon_vals no callback set 2023-09-30T00:53:13.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.097+0000 7fdf2763d700 1 librados: init done 2023-09-30T00:53:13.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.098+0000 7fdf2763d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:13.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.098+0000 7fdf2763d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:13.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.098+0000 7fdf2763d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.40:3301/0 2023-09-30T00:53:13.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.098+0000 7fdf2763d700 1 -- 172.21.15.40:0/4217105522 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf20064a00 con 0x7fdf2013ac20 2023-09-30T00:53:13.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.134+0000 7fdf157fa700 1 -- 172.21.15.40:0/4217105522 <== mon.2 v2:172.21.15.40:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fdf18014620 con 0x7fdf2013ac20 2023-09-30T00:53:13.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.134+0000 7fdf157fa700 1 --2- 172.21.15.40:0/4217105522 >> v2:172.21.15.135:6800/116426 conn(0x7fdf04068910 0x7fdf0406adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:13.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.134+0000 7fdf157fa700 1 -- 172.21.15.40:0/4217105522 <== mon.2 v2:172.21.15.40:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fdf18084620 con 0x7fdf2013ac20 2023-09-30T00:53:13.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.134+0000 7fdf157fa700 10 client.4730.objecter ms_dispatch 0x7fdf20001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:13.135 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.134+0000 7fdf157fa700 3 client.4730.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:13.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.134+0000 7fdf157fa700 3 client.4730.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:13.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.135+0000 7fdf157fa700 20 client.4730.objecter dump_active .. 0 homeless 2023-09-30T00:53:13.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.135+0000 7fdf253d9700 1 --2- 172.21.15.40:0/4217105522 >> v2:172.21.15.135:6800/116426 conn(0x7fdf04068910 0x7fdf0406adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:13.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.135+0000 7fdf253d9700 10 monclient: get_auth_request con 0x7fdf04068910 auth_method 0 2023-09-30T00:53:13.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.135+0000 7fdf157fa700 10 client.4730.objecter ms_handle_connect 0x7fdf04068910 2023-09-30T00:53:13.136 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.135+0000 7fdf253d9700 1 --2- 172.21.15.40:0/4217105522 >> v2:172.21.15.135:6800/116426 conn(0x7fdf04068910 0x7fdf0406adc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c0093f0 tx=0x7fdf1c016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.413+0000 7fdf157fa700 1 -- 172.21.15.40:0/4217105522 <== mon.2 v2:172.21.15.40:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdf18051ec0 con 0x7fdf2013ac20 2023-09-30T00:53:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.413+0000 7fdf157fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:13.413 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.413+0000 7fdf157fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:13.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.632+0000 7fdf2763d700 1 -- 172.21.15.40:0/4217105522 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf200a9d50 con 0x7fdf04068910 2023-09-30T00:53:13.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.633+0000 7fdf157fa700 1 -- 172.21.15.40:0/4217105522 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdf200a9d50 con 0x7fdf04068910 2023-09-30T00:53:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 10 librados: watch_flush enter 2023-09-30T00:53:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 10 librados: watch_flush exit 2023-09-30T00:53:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 20 client.4730.objecter shutdown clearing up homeless session... 2023-09-30T00:53:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 10 client.4730.objecter successfully canceled tick 2023-09-30T00:53:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 1 -- 172.21.15.40:0/4217105522 >> v2:172.21.15.135:6800/116426 conn(0x7fdf04068910 msgr2=0x7fdf0406adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 1 --2- 172.21.15.40:0/4217105522 >> v2:172.21.15.135:6800/116426 conn(0x7fdf04068910 0x7fdf0406adc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c0093f0 tx=0x7fdf1c016000 comp rx=0 tx=0).stop 2023-09-30T00:53:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 10 monclient: shutdown 2023-09-30T00:53:13.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 1 -- 172.21.15.40:0/4217105522 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf2013ac20 msgr2=0x7fdf2014d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 1 --2- 172.21.15.40:0/4217105522 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf2013ac20 0x7fdf2014d8f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdf1800bc30 tx=0x7fdf18009820 comp rx=0 tx=0).stop 2023-09-30T00:53:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 1 -- 172.21.15.40:0/4217105522 shutdown_connections 2023-09-30T00:53:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 1 --2- 172.21.15.40:0/4217105522 >> v2:172.21.15.135:6800/116426 conn(0x7fdf04068910 0x7fdf0406adc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 1 --2- 172.21.15.40:0/4217105522 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fdf200a5fb0 0x7fdf2013dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 1 --2- 172.21.15.40:0/4217105522 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fdf2013ac20 0x7fdf2014d8f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 1 --2- 172.21.15.40:0/4217105522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf200b1460 0x7fdf200a9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 1 -- 172.21.15.40:0/4217105522 >> 172.21.15.40:0/4217105522 conn(0x7fdf200631f0 msgr2=0x7fdf20142870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 1 -- 172.21.15.40:0/4217105522 shutdown_connections 2023-09-30T00:53:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 1 -- 172.21.15.40:0/4217105522 wait complete. 2023-09-30T00:53:13.637 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.635+0000 7fdf2763d700 1 librados: shutdown 2023-09-30T00:53:13.638 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 3 to deep-scrub 2023-09-30T00:53:13.650 INFO:tasks.ceph:Scrubbing osd.4 2023-09-30T00:53:13.650 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:53:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.784+0000 7f37991af700 10 monclient: get_monmap_and_config 2023-09-30T00:53:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.784+0000 7f37991af700 10 monclient: build_initial_monmap 2023-09-30T00:53:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.784+0000 7f37991af700 10 monclient: monmap: 2023-09-30T00:53:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:13.785755+0000 2023-09-30T00:53:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:13.785755+0000 2023-09-30T00:53:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:13.787 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:13.787 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:13.787 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:13.787 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.785+0000 7f37991af700 1 Processor -- start 2023-09-30T00:53:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.785+0000 7f37991af700 1 -- start start 2023-09-30T00:53:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.785+0000 7f37991af700 10 monclient: init 2023-09-30T00:53:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.785+0000 7f37991af700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.785+0000 7f37991af700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-09-30T00:53:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.785+0000 7f37991af700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37940a5710 addr v1:172.21.15.40:6789/0 2023-09-30T00:53:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.786+0000 7f37991af700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f37940a7630 0x7f37940a7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.786+0000 7f37991af700 10 monclient(hunting): picked mon.noname-a con 0x7f37940a7630 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.786+0000 7f37991af700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f37940a7fc0 0x7f379413af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.786+0000 7f37991af700 10 monclient(hunting): picked mon.noname-c con 0x7f37940a7fc0 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.786+0000 7f37991af700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37940b1770 con 0x7f37940a5710 2023-09-30T00:53:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.786+0000 7f37991af700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:13.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.786+0000 7f37991af700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37940a5ae0 con 0x7f37940a7630 2023-09-30T00:53:13.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.786+0000 7f37991af700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:13.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.786+0000 7f37991af700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37940b1440 con 0x7f37940a7fc0 2023-09-30T00:53:13.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.786+0000 7f37991af700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:13.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.786+0000 7f37991af700 10 monclient(hunting): authenticate will time out at 5655.087081s 2023-09-30T00:53:13.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.786+0000 7f379359e700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f37940a7fc0 0x7f379413af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:13.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.787+0000 7f379259c700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f37940a7630 0x7f37940a7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:13.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.787+0000 7f3792d9d700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f37940a5710 0x7f37940b0b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53304/0 (socket says 172.21.15.40:53304) 2023-09-30T00:53:13.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.787+0000 7f3792d9d700 1 -- 172.21.15.40:0/883210507 learned_addr learned my addr 172.21.15.40:0/883210507 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:53:13.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.787+0000 7f379359e700 10 monclient(hunting): get_auth_request con 0x7f37940a7fc0 auth_method 0 2023-09-30T00:53:13.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.787+0000 7f379359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:13.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.787+0000 7f379359e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:13.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.787+0000 7f379359e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:13.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.787+0000 7f379259c700 10 monclient(hunting): get_auth_request con 0x7f37940a7630 auth_method 0 2023-09-30T00:53:13.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.787+0000 7f379259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:13.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.787+0000 7f379259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:13.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.787+0000 7f379259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:13.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.787+0000 7f379359e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:13.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.787+0000 7f379359e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:13.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.787+0000 7f379359e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:13.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.788+0000 7f379259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:13.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.788+0000 7f379259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:13.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.788+0000 7f379259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:13.793 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.789+0000 7f379359e700 10 monclient(hunting): handle_auth_done global_id 4945 payload 274 2023-09-30T00:53:13.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.789+0000 7f379359e700 1 -- 172.21.15.40:0/883210507 >> v2:172.21.15.40:3300/0 conn(0x7f37940a7630 msgr2=0x7f37940a7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:13.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.789+0000 7f379359e700 1 --2- 172.21.15.40:0/883210507 >> v2:172.21.15.40:3300/0 conn(0x7f37940a7630 0x7f37940a7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.789+0000 7f379359e700 1 -- 172.21.15.40:0/883210507 >> v1:172.21.15.40:6789/0 conn(0x7f37940a5710 legacy=0x7f37940b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:13.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.789+0000 7f379359e700 10 monclient: _finish_hunting 0 2023-09-30T00:53:13.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.789+0000 7f379359e700 1 monclient: found mon.noname-c 2023-09-30T00:53:13.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.789+0000 7f379359e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:13.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.789+0000 7f379359e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:53:13.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.789+0000 7f379359e700 1 -- 172.21.15.40:0/883210507 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37940a7210 con 0x7f37940a7fc0 2023-09-30T00:53:13.794 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.791+0000 7f379359e700 1 --2- 172.21.15.40:0/883210507 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37940a7fc0 0x7f379413af10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f37940b16e0 tx=0x7f378c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c438a41f345ea7b9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:13.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f3791d9b700 1 -- 172.21.15.40:0/883210507 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c00e070 con 0x7f37940a7fc0 2023-09-30T00:53:13.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f3791d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:13.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f3791d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:53:13.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f3791d9b700 10 monclient: dump: 2023-09-30T00:53:13.796 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:13.796 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:13.797 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.797 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.797 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:13.797 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:13.797 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:13.797 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:13.797 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:13.797 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:13.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f3791d9b700 10 monclient: _finish_auth 0 2023-09-30T00:53:13.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f3791d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:13.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f3791d9b700 1 -- 172.21.15.40:0/883210507 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378c00bb60 con 0x7f37940a7fc0 2023-09-30T00:53:13.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f37991af700 5 monclient: authenticate success, global_id 4945 2023-09-30T00:53:13.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f37991af700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:13.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f3791d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:13.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f37991af700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:13.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f3791d9b700 1 -- 172.21.15.40:0/883210507 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c012430 con 0x7f37940a7fc0 2023-09-30T00:53:13.798 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f37991af700 4 set_mon_vals no callback set 2023-09-30T00:53:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f3791d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f3791d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.795+0000 7f3791d9b700 10 monclient: dump: 2023-09-30T00:53:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:13.799 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:13.800 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:13.800 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:13.800 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:13.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.796+0000 7f37991af700 10 monclient: shutdown 2023-09-30T00:53:13.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.796+0000 7f37991af700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:13.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.796+0000 7f37991af700 1 -- 172.21.15.40:0/883210507 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37940a7fc0 msgr2=0x7f379413af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:13.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.796+0000 7f37991af700 1 --2- 172.21.15.40:0/883210507 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37940a7fc0 0x7f379413af10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f37940b16e0 tx=0x7f378c009780 comp rx=0 tx=0).stop 2023-09-30T00:53:13.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.796+0000 7f37991af700 1 -- 172.21.15.40:0/883210507 shutdown_connections 2023-09-30T00:53:13.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.796+0000 7f37991af700 1 --2- 172.21.15.40:0/883210507 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37940a7fc0 0x7f379413af10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.803 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.796+0000 7f37991af700 1 --2- 172.21.15.40:0/883210507 >> v2:172.21.15.40:3300/0 conn(0x7f37940a7630 0x7f37940a7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.796+0000 7f37991af700 1 -- 172.21.15.40:0/883210507 >> 172.21.15.40:0/883210507 conn(0x7f3794069060 msgr2=0x7f37940631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.796+0000 7f37991af700 1 -- 172.21.15.40:0/883210507 shutdown_connections 2023-09-30T00:53:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.796+0000 7f37991af700 1 -- 172.21.15.40:0/883210507 wait complete. 2023-09-30T00:53:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.797+0000 7f37991af700 10 monclient: build_initial_monmap 2023-09-30T00:53:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.797+0000 7f37991af700 10 monclient: monmap: 2023-09-30T00:53:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:13.804 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:13.805 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.797+0000 7f37991af700 1 librados: starting msgr at 2023-09-30T00:53:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.797+0000 7f37991af700 1 librados: starting objecter 2023-09-30T00:53:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.797+0000 7f37991af700 1 Processor -- start 2023-09-30T00:53:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.797+0000 7f37991af700 1 -- start start 2023-09-30T00:53:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.797+0000 7f37991af700 1 librados: setting wanted keys 2023-09-30T00:53:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.797+0000 7f37991af700 1 librados: calling monclient init 2023-09-30T00:53:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.797+0000 7f37991af700 10 monclient: init 2023-09-30T00:53:13.805 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:13.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:53:13.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37940a5710 0x7f379413ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:13.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 10 monclient(hunting): picked mon.noname-b con 0x7f37940a5710 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:13.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37940a7630 0x7f379414e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:13.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 10 monclient(hunting): picked mon.noname-c con 0x7f37940a7630 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:13.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37940a7fc0 0x7f37940a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:13.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 10 monclient(hunting): picked mon.noname-a con 0x7f37940a7fc0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:13.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:13.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37940aa290 con 0x7f37940a7fc0 2023-09-30T00:53:13.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:13.806 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379414dc00 con 0x7f37940a5710 2023-09-30T00:53:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37940a89f0 con 0x7f37940a7630 2023-09-30T00:53:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f37991af700 10 monclient(hunting): authenticate will time out at 5655.099394s 2023-09-30T00:53:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f379359e700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37940a7fc0 0x7f37940a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.798+0000 7f3792d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37940a5710 0x7f379413ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f379259c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37940a7630 0x7f379414e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f3792d9d700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37940a5710 0x7f379413ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:56532/0 (socket says 172.21.15.40:56532) 2023-09-30T00:53:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f3792d9d700 1 -- 172.21.15.40:0/3588848738 learned_addr learned my addr 172.21.15.40:0/3588848738 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:13.807 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f3792d9d700 10 monclient(hunting): get_auth_request con 0x7f37940a5710 auth_method 0 2023-09-30T00:53:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f3792d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f3792d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f3792d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f379359e700 10 monclient(hunting): get_auth_request con 0x7f37940a7fc0 auth_method 0 2023-09-30T00:53:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f379359e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f379359e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f379359e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f379259c700 10 monclient(hunting): get_auth_request con 0x7f37940a7630 auth_method 0 2023-09-30T00:53:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f379259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f379259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f379259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:13.808 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f3792d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f3792d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f3792d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f379359e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f379359e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.799+0000 7f379359e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.800+0000 7f379259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.800+0000 7f379259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.800+0000 7f379259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.802+0000 7f3792d9d700 10 monclient(hunting): handle_auth_done global_id 4948 payload 824 2023-09-30T00:53:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.802+0000 7f3792d9d700 1 -- 172.21.15.40:0/3588848738 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37940a7630 msgr2=0x7f379414e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.802+0000 7f3792d9d700 1 --2- 172.21.15.40:0/3588848738 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f37940a7630 0x7f379414e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.809 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.802+0000 7f3792d9d700 1 -- 172.21.15.40:0/3588848738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37940a7fc0 msgr2=0x7f37940a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:13.810 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.802+0000 7f3792d9d700 1 --2- 172.21.15.40:0/3588848738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37940a7fc0 0x7f37940a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.810 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.802+0000 7f3792d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:53:13.810 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.802+0000 7f3792d9d700 1 monclient: found mon.noname-b 2023-09-30T00:53:13.810 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.802+0000 7f3792d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:13.810 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.802+0000 7f3792d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:13.810 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.802+0000 7f3792d9d700 1 -- 172.21.15.40:0/3588848738 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c009a20 con 0x7f37940a5710 2023-09-30T00:53:13.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.805+0000 7f3792d9d700 1 --2- 172.21.15.40:0/3588848738 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37940a5710 0x7f379413ecd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f378400c7c0 tx=0x7f3784007f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:13.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.805+0000 7f378affd700 10 client.?.objecter ms_handle_connect 0x7f37940a5710 2023-09-30T00:53:13.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.805+0000 7f378affd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:13.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.808+0000 7f378affd700 1 -- 172.21.15.40:0/3588848738 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3784005b30 con 0x7f37940a5710 2023-09-30T00:53:13.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.808+0000 7f378affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:13.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.808+0000 7f378affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:13.812 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.808+0000 7f378affd700 10 monclient: dump: 2023-09-30T00:53:13.812 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:13.812 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:13.812 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:13.813 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f378affd700 10 monclient: _finish_auth 0 2023-09-30T00:53:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f378affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f378affd700 1 -- 172.21.15.40:0/3588848738 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3784005c90 con 0x7f37940a5710 2023-09-30T00:53:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 5 monclient: authenticate success, global_id 4948 2023-09-30T00:53:13.813 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 10 monclient: _renew_subs 2023-09-30T00:53:13.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:13.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 1 -- 172.21.15.40:0/3588848738 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37940a9350 con 0x7f37940a5710 2023-09-30T00:53:13.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f378affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:13.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f378affd700 1 -- 172.21.15.40:0/3588848738 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3784007320 con 0x7f37940a5710 2023-09-30T00:53:13.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f378affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:13.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f378affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:13.814 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f378affd700 10 monclient: dump: 2023-09-30T00:53:13.814 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:13.814 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:13.814 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.814 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:13.815 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f379159a700 4 set_mon_vals no callback set 2023-09-30T00:53:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 10 client.4948.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 10 monclient: _renew_subs 2023-09-30T00:53:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 1 -- 172.21.15.40:0/3588848738 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379414d930 con 0x7f37940a5710 2023-09-30T00:53:13.815 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 1 librados: init done 2023-09-30T00:53:13.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 10 client.4948.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:13.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 15 client.4948.objecter _session_command_op_assign -1 1 2023-09-30T00:53:13.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 20 client.4948.objecter _get_session osd=-1 returning homeless 2023-09-30T00:53:13.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 10 client.4948.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:13.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:13.816 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.809+0000 7f37991af700 1 -- 172.21.15.40:0/3588848738 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3794064a00 con 0x7f37940a5710 2023-09-30T00:53:13.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.829+0000 7f378affd700 1 -- 172.21.15.40:0/3588848738 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3784007690 con 0x7f37940a5710 2023-09-30T00:53:13.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.829+0000 7f378affd700 1 --2- 172.21.15.40:0/3588848738 >> v2:172.21.15.135:6800/116426 conn(0x7f3774068960 0x7f377406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:13.829 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.829+0000 7f379259c700 1 --2- 172.21.15.40:0/3588848738 >> v2:172.21.15.135:6800/116426 conn(0x7f3774068960 0x7f377406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:13.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.829+0000 7f379259c700 10 monclient: get_auth_request con 0x7f3774068960 auth_method 0 2023-09-30T00:53:13.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.830+0000 7f378affd700 10 client.4948.objecter ms_handle_connect 0x7f3774068960 2023-09-30T00:53:13.830 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.830+0000 7f379259c700 1 --2- 172.21.15.40:0/3588848738 >> v2:172.21.15.135:6800/116426 conn(0x7f3774068960 0x7f377406ae10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f379414e800 tx=0x7f377c0091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:13.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.833+0000 7f378affd700 1 -- 172.21.15.40:0/3588848738 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f37840556b0 con 0x7f37940a5710 2023-09-30T00:53:13.834 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.833+0000 7f378affd700 10 client.4948.objecter ms_dispatch 0x7f3794001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:13.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.833+0000 7f378affd700 3 client.4948.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:13.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.833+0000 7f378affd700 3 client.4948.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:13.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f378affd700 10 client.4948.objecter checking command 1 2023-09-30T00:53:13.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f378affd700 1 --2- 172.21.15.40:0/3588848738 >> v2:172.21.15.135:6805/117131 conn(0x7f377406e8d0 0x7f3774070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:13.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f378affd700 20 client.4948.objecter _get_session s=0x7f377406e220 osd=4 3 2023-09-30T00:53:13.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f378affd700 20 client.4948.objecter put_session s=0x7f377406e220 osd=4 3 2023-09-30T00:53:13.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f378affd700 15 client.4948.objecter _session_command_op_remove -1 1 2023-09-30T00:53:13.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f378affd700 20 client.4948.objecter _get_session s=0x7f377406e220 osd=4 3 2023-09-30T00:53:13.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f378affd700 20 client.4948.objecter get_session s=0x7f377406e220 osd=4 3 2023-09-30T00:53:13.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f378affd700 15 client.4948.objecter _session_command_op_assign 4 1 2023-09-30T00:53:13.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f378affd700 20 client.4948.objecter put_session s=0x7f377406e220 osd=4 4 2023-09-30T00:53:13.835 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f378affd700 10 client.4948.objecter _send_command 1 2023-09-30T00:53:13.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f378affd700 1 -- 172.21.15.40:0/3588848738 --> v2:172.21.15.135:6805/117131 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3774071350 con 0x7f377406e8d0 2023-09-30T00:53:13.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f378affd700 20 client.4948.objecter dump_active .. 0 homeless 2023-09-30T00:53:13.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f378affd700 1 -- 172.21.15.40:0/3588848738 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37840512f0 con 0x7f37940a5710 2023-09-30T00:53:13.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f378affd700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:53:13.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.834+0000 7f379359e700 1 --2- 172.21.15.40:0/3588848738 >> v2:172.21.15.135:6805/117131 conn(0x7f377406e8d0 0x7f3774070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:13.836 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.835+0000 7f379359e700 10 monclient: get_auth_request con 0x7f377406e8d0 auth_method 0 2023-09-30T00:53:13.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.838+0000 7f379359e700 1 --2- 172.21.15.40:0/3588848738 >> v2:172.21.15.135:6805/117131 conn(0x7f377406e8d0 0x7f3774070ce0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:13.838 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.838+0000 7f378affd700 10 client.4948.objecter ms_handle_connect 0x7f377406e8d0 2023-09-30T00:53:13.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.862+0000 7f378affd700 1 -- 172.21.15.40:0/3588848738 <== osd.4 v2:172.21.15.135:6805/117131 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f3774071350 con 0x7f377406e8d0 2023-09-30T00:53:13.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.862+0000 7f378affd700 10 client.4948.objecter ms_dispatch 0x7f3794001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:53:13.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.862+0000 7f378affd700 10 client.4948.objecter _finish_command 1 = system:0 2023-09-30T00:53:13.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.862+0000 7f378affd700 20 client.4948.objecter put_session s=0x7f377406e220 osd=4 4 2023-09-30T00:53:13.863 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.862+0000 7f378affd700 15 client.4948.objecter _session_command_op_remove 4 1 2023-09-30T00:53:13.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.893+0000 7f37991af700 10 client.4948.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T00:53:13.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.893+0000 7f37991af700 15 client.4948.objecter _session_command_op_assign -1 2 2023-09-30T00:53:13.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.893+0000 7f37991af700 20 client.4948.objecter _get_session s=0x7f377406e220 osd=4 3 2023-09-30T00:53:13.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.893+0000 7f37991af700 20 client.4948.objecter put_session s=0x7f377406e220 osd=4 3 2023-09-30T00:53:13.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.893+0000 7f37991af700 20 client.4948.objecter _get_session s=0x7f377406e220 osd=4 3 2023-09-30T00:53:13.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.893+0000 7f37991af700 15 client.4948.objecter _session_command_op_remove -1 2 2023-09-30T00:53:13.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.893+0000 7f37991af700 20 client.4948.objecter get_session s=0x7f377406e220 osd=4 3 2023-09-30T00:53:13.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.893+0000 7f37991af700 15 client.4948.objecter _session_command_op_assign 4 2 2023-09-30T00:53:13.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.893+0000 7f37991af700 20 client.4948.objecter put_session s=0x7f377406e220 osd=4 4 2023-09-30T00:53:13.894 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.893+0000 7f37991af700 10 client.4948.objecter _send_command 2 2023-09-30T00:53:13.895 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.893+0000 7f37991af700 1 -- 172.21.15.40:0/3588848738 --> v2:172.21.15.135:6805/117131 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f379413b450 con 0x7f377406e8d0 2023-09-30T00:53:13.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f378affd700 1 -- 172.21.15.40:0/3588848738 <== osd.4 v2:172.21.15.135:6805/117131 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f379413b450 con 0x7f377406e8d0 2023-09-30T00:53:13.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f378affd700 10 client.4948.objecter ms_dispatch 0x7f3794001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:53:13.917 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f378affd700 10 client.4948.objecter _finish_command 2 = system:0 2023-09-30T00:53:13.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f378affd700 20 client.4948.objecter put_session s=0x7f377406e220 osd=4 4 2023-09-30T00:53:13.918 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f378affd700 15 client.4948.objecter _session_command_op_remove 4 2 2023-09-30T00:53:13.918 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-09-30T00:53:13.918 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:53:13.918 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-09-30T00:53:13.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 10 librados: watch_flush enter 2023-09-30T00:53:13.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 10 librados: watch_flush exit 2023-09-30T00:53:13.919 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 10 client.4948.objecter close_session for osd.4 2023-09-30T00:53:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 1 -- 172.21.15.40:0/3588848738 >> v2:172.21.15.135:6805/117131 conn(0x7f377406e8d0 msgr2=0x7f3774070ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 1 --2- 172.21.15.40:0/3588848738 >> v2:172.21.15.135:6805/117131 conn(0x7f377406e8d0 0x7f3774070ce0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 20 client.4948.objecter put_session s=0x7f377406e220 osd=4 1 2023-09-30T00:53:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 20 client.4948.objecter shutdown clearing up homeless session... 2023-09-30T00:53:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 10 client.4948.objecter successfully canceled tick 2023-09-30T00:53:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 1 -- 172.21.15.40:0/3588848738 >> v2:172.21.15.135:6800/116426 conn(0x7f3774068960 msgr2=0x7f377406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 1 --2- 172.21.15.40:0/3588848738 >> v2:172.21.15.135:6800/116426 conn(0x7f3774068960 0x7f377406ae10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f379414e800 tx=0x7f377c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 10 monclient: shutdown 2023-09-30T00:53:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 1 -- 172.21.15.40:0/3588848738 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37940a5710 msgr2=0x7f379413ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:13.920 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 1 --2- 172.21.15.40:0/3588848738 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f37940a5710 0x7f379413ecd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f378400c7c0 tx=0x7f3784007f30 comp rx=0 tx=0).stop 2023-09-30T00:53:13.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f379359e700 1 -- 172.21.15.40:0/3588848738 reap_dead start 2023-09-30T00:53:13.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 1 -- 172.21.15.40:0/3588848738 shutdown_connections 2023-09-30T00:53:13.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 1 -- 172.21.15.40:0/3588848738 >> 172.21.15.40:0/3588848738 conn(0x7f3794069060 msgr2=0x7f3794141f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:13.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 1 -- 172.21.15.40:0/3588848738 shutdown_connections 2023-09-30T00:53:13.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.917+0000 7f37991af700 1 -- 172.21.15.40:0/3588848738 wait complete. 2023-09-30T00:53:13.921 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:13.918+0000 7f37991af700 1 librados: shutdown 2023-09-30T00:53:13.935 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-09-30T00:53:14.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.069+0000 7f24f28f6700 10 monclient: get_monmap_and_config 2023-09-30T00:53:14.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.069+0000 7f24f28f6700 10 monclient: build_initial_monmap 2023-09-30T00:53:14.070 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.070+0000 7f24f28f6700 10 monclient: monmap: 2023-09-30T00:53:14.071 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:14.071 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:14.071 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:14.071034+0000 2023-09-30T00:53:14.071 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:14.071034+0000 2023-09-30T00:53:14.071 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:14.071 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:14.071 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:14.071 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:14.071 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:14.071 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:14.071 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:14.071 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:14.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.070+0000 7f24f28f6700 1 Processor -- start 2023-09-30T00:53:14.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.070+0000 7f24f28f6700 1 -- start start 2023-09-30T00:53:14.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.070+0000 7f24f28f6700 10 monclient: init 2023-09-30T00:53:14.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.071+0000 7f24f28f6700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:14.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.071+0000 7f24f28f6700 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-09-30T00:53:14.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.071+0000 7f24f28f6700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f24ec0b1460 0x7f24ec0b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:14.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.071+0000 7f24f28f6700 10 monclient(hunting): picked mon.noname-a con 0x7f24ec0b1460 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:14.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.071+0000 7f24f28f6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f24ec0a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:14.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.071+0000 7f24f28f6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24ec13ac20 addr v1:172.21.15.40:6789/0 2023-09-30T00:53:14.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.071+0000 7f24f28f6700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ec0a5b20 con 0x7f24ec13ac20 2023-09-30T00:53:14.073 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.071+0000 7f24f28f6700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ec0a71f0 con 0x7f24ec0a5fb0 2023-09-30T00:53:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.071+0000 7f24f28f6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.071+0000 7f24f28f6700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24ec0a74d0 con 0x7f24ec0b1460 2023-09-30T00:53:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.071+0000 7f24f28f6700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.071+0000 7f24f28f6700 10 monclient(hunting): authenticate will time out at 5655.372525s 2023-09-30T00:53:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.072+0000 7f24ebfff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f24ec0b1460 0x7f24ec0b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.072+0000 7f24f0e93700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f24ec13ac20 0x7f24ec13b080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53314/0 (socket says 172.21.15.40:53314) 2023-09-30T00:53:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.072+0000 7f24f0e93700 1 -- 172.21.15.40:0/3516984996 learned_addr learned my addr 172.21.15.40:0/3516984996 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:53:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.073+0000 7f24ebfff700 10 monclient(hunting): get_auth_request con 0x7f24ec0b1460 auth_method 0 2023-09-30T00:53:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.073+0000 7f24ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.073+0000 7f24ebfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.073+0000 7f24ebfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.074+0000 7f24ebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.074+0000 7f24ebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.074+0000 7f24ebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:14.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.076+0000 7f24eaffd700 1 -- 172.21.15.40:0/3516984996 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708123152 0 0) 0x7f24ec0a71f0 con 0x7f24ec0a5fb0 2023-09-30T00:53:14.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.076+0000 7f24eaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:14.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.076+0000 7f24eaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:14.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.076+0000 7f24eaffd700 10 monclient(hunting): my global_id is 4951 2023-09-30T00:53:14.077 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.076+0000 7f24eaffd700 1 -- 172.21.15.40:0/3516984996 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d4003570 con 0x7f24ec0a5fb0 2023-09-30T00:53:14.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.079+0000 7f24ebfff700 10 monclient(hunting): handle_auth_done global_id 4866 payload 274 2023-09-30T00:53:14.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.079+0000 7f24ebfff700 1 -- 172.21.15.40:0/3516984996 >> v1:172.21.15.135:6789/0 conn(0x7f24ec0a5fb0 legacy=0x7f24ec141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:14.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.079+0000 7f24ebfff700 1 -- 172.21.15.40:0/3516984996 >> v1:172.21.15.40:6789/0 conn(0x7f24ec13ac20 legacy=0x7f24ec13b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:14.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.079+0000 7f24ebfff700 10 monclient: _finish_hunting 0 2023-09-30T00:53:14.079 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.079+0000 7f24ebfff700 1 monclient: found mon.noname-a 2023-09-30T00:53:14.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.079+0000 7f24ebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:14.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.079+0000 7f24ebfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:53:14.080 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.079+0000 7f24ebfff700 1 -- 172.21.15.40:0/3516984996 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec0a63c0 con 0x7f24ec0b1460 2023-09-30T00:53:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.082+0000 7f24ebfff700 1 --2- 172.21.15.40:0/3516984996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24ec0b1460 0x7f24ec0b1830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009910 tx=0x7f24dc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0db720a0ae3790 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:14.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24eaffd700 1 -- 172.21.15.40:0/3516984996 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc00b640 con 0x7f24ec0b1460 2023-09-30T00:53:14.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24eaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:14.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24eaffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:53:14.091 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24eaffd700 10 monclient: dump: 2023-09-30T00:53:14.091 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:14.091 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:14.091 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.091 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.091 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:14.091 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:14.092 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24eaffd700 10 monclient: _finish_auth 0 2023-09-30T00:53:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24eaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24eaffd700 1 -- 172.21.15.40:0/3516984996 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24dc00b7a0 con 0x7f24ec0b1460 2023-09-30T00:53:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24eaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24eaffd700 1 -- 172.21.15.40:0/3516984996 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc00bb60 con 0x7f24ec0b1460 2023-09-30T00:53:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24eaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24eaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:53:14.092 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24eaffd700 10 monclient: dump: 2023-09-30T00:53:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:14.093 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24f28f6700 5 monclient: authenticate success, global_id 4866 2023-09-30T00:53:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24f28f6700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.090+0000 7f24f28f6700 4 set_mon_vals no callback set 2023-09-30T00:53:14.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.091+0000 7f24f28f6700 10 monclient: shutdown 2023-09-30T00:53:14.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.091+0000 7f24f28f6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:14.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.091+0000 7f24f28f6700 1 -- 172.21.15.40:0/3516984996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24ec0b1460 msgr2=0x7f24ec0b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:14.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.091+0000 7f24f28f6700 1 --2- 172.21.15.40:0/3516984996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24ec0b1460 0x7f24ec0b1830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f24dc009910 tx=0x7f24dc0091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.091+0000 7f24f28f6700 1 -- 172.21.15.40:0/3516984996 shutdown_connections 2023-09-30T00:53:14.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.091+0000 7f24f28f6700 1 --2- 172.21.15.40:0/3516984996 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24ec0b1460 0x7f24ec0b1830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.091+0000 7f24f28f6700 1 -- 172.21.15.40:0/3516984996 >> 172.21.15.40:0/3516984996 conn(0x7f24ec0631f0 msgr2=0x7f24ec0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:14.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.091+0000 7f24f28f6700 1 -- 172.21.15.40:0/3516984996 shutdown_connections 2023-09-30T00:53:14.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.091+0000 7f24f28f6700 1 -- 172.21.15.40:0/3516984996 wait complete. 2023-09-30T00:53:14.097 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.092+0000 7f24f28f6700 10 monclient: build_initial_monmap 2023-09-30T00:53:14.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.092+0000 7f24f28f6700 10 monclient: monmap: 2023-09-30T00:53:14.098 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:14.098 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:14.098 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:14.098 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:14.098 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:14.098 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:14.098 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:14.098 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:14.098 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:14.098 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:14.098 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.092+0000 7f24f28f6700 1 librados: starting msgr at 2023-09-30T00:53:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.092+0000 7f24f28f6700 1 librados: starting objecter 2023-09-30T00:53:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.092+0000 7f24f28f6700 1 Processor -- start 2023-09-30T00:53:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.092+0000 7f24f28f6700 1 -- start start 2023-09-30T00:53:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.092+0000 7f24f28f6700 1 librados: setting wanted keys 2023-09-30T00:53:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.092+0000 7f24f28f6700 1 librados: calling monclient init 2023-09-30T00:53:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.092+0000 7f24f28f6700 10 monclient: init 2023-09-30T00:53:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:53:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24ec0a5fb0 0x7f24ec13e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 10 monclient(hunting): picked mon.noname-c con 0x7f24ec0a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24ec0b1460 0x7f24ec0a79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:14.099 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 10 monclient(hunting): picked mon.noname-b con 0x7f24ec0b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:14.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24ec13ac20 0x7f24ec1510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:14.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 10 monclient(hunting): picked mon.noname-a con 0x7f24ec13ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:14.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:14.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec0ab520 con 0x7f24ec13ac20 2023-09-30T00:53:14.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:14.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec0a9ed0 con 0x7f24ec0b1460 2023-09-30T00:53:14.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:14.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec0a9bd0 con 0x7f24ec0a5fb0 2023-09-30T00:53:14.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:14.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.093+0000 7f24f28f6700 10 monclient(hunting): authenticate will time out at 5655.394504s 2023-09-30T00:53:14.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24ebfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24ec0a5fb0 0x7f24ec13e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:14.100 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24f0e93700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24ec13ac20 0x7f24ec1510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24eb7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24ec0b1460 0x7f24ec0a79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24ebfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24ec0a5fb0 0x7f24ec13e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:37548/0 (socket says 172.21.15.40:37548) 2023-09-30T00:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24ebfff700 1 -- 172.21.15.40:0/2593945652 learned_addr learned my addr 172.21.15.40:0/2593945652 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24ebfff700 10 monclient(hunting): get_auth_request con 0x7f24ec0a5fb0 auth_method 0 2023-09-30T00:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24ebfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24ebfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24f0e93700 10 monclient(hunting): get_auth_request con 0x7f24ec13ac20 auth_method 0 2023-09-30T00:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24f0e93700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24f0e93700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24f0e93700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24eb7fe700 10 monclient(hunting): get_auth_request con 0x7f24ec0b1460 auth_method 0 2023-09-30T00:53:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24eb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24eb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.094+0000 7f24eb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.095+0000 7f24ebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.095+0000 7f24ebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.095+0000 7f24ebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.095+0000 7f24eb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.095+0000 7f24eb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.095+0000 7f24eb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.095+0000 7f24f0e93700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:14.102 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.095+0000 7f24f0e93700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.095+0000 7f24f0e93700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.098+0000 7f24eb7fe700 10 monclient(hunting): handle_auth_done global_id 4954 payload 824 2023-09-30T00:53:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.098+0000 7f24eb7fe700 1 -- 172.21.15.40:0/2593945652 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24ec0a5fb0 msgr2=0x7f24ec13e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.098+0000 7f24eb7fe700 1 --2- 172.21.15.40:0/2593945652 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24ec0a5fb0 0x7f24ec13e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.098+0000 7f24eb7fe700 1 -- 172.21.15.40:0/2593945652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24ec13ac20 msgr2=0x7f24ec1510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.098+0000 7f24eb7fe700 1 --2- 172.21.15.40:0/2593945652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24ec13ac20 0x7f24ec1510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.098+0000 7f24eb7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:53:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.098+0000 7f24eb7fe700 1 monclient: found mon.noname-b 2023-09-30T00:53:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.098+0000 7f24eb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.098+0000 7f24eb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.098+0000 7f24eb7fe700 1 -- 172.21.15.40:0/2593945652 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc009420 con 0x7f24ec0b1460 2023-09-30T00:53:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.100+0000 7f24eb7fe700 1 --2- 172.21.15.40:0/2593945652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24ec0b1460 0x7f24ec0a79f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f24e000c560 tx=0x7f24e0007d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:14.104 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.100+0000 7f24d3fff700 10 client.?.objecter ms_handle_connect 0x7f24ec0b1460 2023-09-30T00:53:14.105 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.100+0000 7f24d3fff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24d3fff700 1 -- 172.21.15.40:0/2593945652 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e000ce90 con 0x7f24ec0b1460 2023-09-30T00:53:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24d3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24d3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24d3fff700 10 monclient: dump: 2023-09-30T00:53:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.106 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24d3fff700 10 monclient: _finish_auth 0 2023-09-30T00:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24d3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24d3fff700 1 -- 172.21.15.40:0/2593945652 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e0011590 con 0x7f24ec0b1460 2023-09-30T00:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24d3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24d3fff700 1 -- 172.21.15.40:0/2593945652 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0011950 con 0x7f24ec0b1460 2023-09-30T00:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24d3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:14.107 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24d3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24d3fff700 10 monclient: dump: 2023-09-30T00:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:14.108 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24f28f6700 5 monclient: authenticate success, global_id 4954 2023-09-30T00:53:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24f28f6700 10 monclient: _renew_subs 2023-09-30T00:53:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24f28f6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24f28f6700 1 -- 172.21.15.40:0/2593945652 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ec13ece0 con 0x7f24ec0b1460 2023-09-30T00:53:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.104+0000 7f24ea7fc700 4 set_mon_vals no callback set 2023-09-30T00:53:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.105+0000 7f24f28f6700 10 client.4954.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.105+0000 7f24f28f6700 10 monclient: _renew_subs 2023-09-30T00:53:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.105+0000 7f24f28f6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.105+0000 7f24f28f6700 1 -- 172.21.15.40:0/2593945652 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ec0a9760 con 0x7f24ec0b1460 2023-09-30T00:53:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.105+0000 7f24f28f6700 1 librados: init done 2023-09-30T00:53:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.105+0000 7f24f28f6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.105+0000 7f24f28f6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:14.109 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.105+0000 7f24f28f6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:14.110 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.105+0000 7f24f28f6700 1 -- 172.21.15.40:0/2593945652 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ec0aa080 con 0x7f24ec0b1460 2023-09-30T00:53:14.325 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.325+0000 7f24d3fff700 1 -- 172.21.15.40:0/2593945652 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f24e0011d40 con 0x7f24ec0b1460 2023-09-30T00:53:14.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.325+0000 7f24d3fff700 1 --2- 172.21.15.40:0/2593945652 >> v2:172.21.15.135:6800/116426 conn(0x7f24d8068960 0x7f24d806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:14.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.325+0000 7f24ebfff700 1 --2- 172.21.15.40:0/2593945652 >> v2:172.21.15.135:6800/116426 conn(0x7f24d8068960 0x7f24d806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:14.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.325+0000 7f24ebfff700 10 monclient: get_auth_request con 0x7f24d8068960 auth_method 0 2023-09-30T00:53:14.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.326+0000 7f24d3fff700 10 client.4954.objecter ms_handle_connect 0x7f24d8068960 2023-09-30T00:53:14.326 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.326+0000 7f24ebfff700 1 --2- 172.21.15.40:0/2593945652 >> v2:172.21.15.135:6800/116426 conn(0x7f24d8068960 0x7f24d806ae10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0091e0 tx=0x7f24dc017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.331+0000 7f24d3fff700 1 -- 172.21.15.40:0/2593945652 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f24e0019310 con 0x7f24ec0b1460 2023-09-30T00:53:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.331+0000 7f24d3fff700 10 client.4954.objecter ms_dispatch 0x7f24ec001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.331+0000 7f24d3fff700 3 client.4954.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.331+0000 7f24d3fff700 3 client.4954.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:14.331 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.331+0000 7f24d3fff700 20 client.4954.objecter dump_active .. 0 homeless 2023-09-30T00:53:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.372+0000 7f24d3fff700 1 -- 172.21.15.40:0/2593945652 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f24e0051ca0 con 0x7f24ec0b1460 2023-09-30T00:53:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.372+0000 7f24d3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:14.372 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.372+0000 7f24d3fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:14.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.587+0000 7f24f28f6700 1 -- 172.21.15.40:0/2593945652 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f24ec0aa080 con 0x7f24d8068960 2023-09-30T00:53:14.588 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.588+0000 7f24d3fff700 1 -- 172.21.15.40:0/2593945652 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f24ec0aa080 con 0x7f24d8068960 2023-09-30T00:53:14.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 10 librados: watch_flush enter 2023-09-30T00:53:14.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 10 librados: watch_flush exit 2023-09-30T00:53:14.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 20 client.4954.objecter shutdown clearing up homeless session... 2023-09-30T00:53:14.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 10 client.4954.objecter successfully canceled tick 2023-09-30T00:53:14.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 1 -- 172.21.15.40:0/2593945652 >> v2:172.21.15.135:6800/116426 conn(0x7f24d8068960 msgr2=0x7f24d806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:14.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 1 --2- 172.21.15.40:0/2593945652 >> v2:172.21.15.135:6800/116426 conn(0x7f24d8068960 0x7f24d806ae10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0091e0 tx=0x7f24dc017000 comp rx=0 tx=0).stop 2023-09-30T00:53:14.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 10 monclient: shutdown 2023-09-30T00:53:14.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:14.591 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 1 -- 172.21.15.40:0/2593945652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24ec0b1460 msgr2=0x7f24ec0a79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:14.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 1 --2- 172.21.15.40:0/2593945652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24ec0b1460 0x7f24ec0a79f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f24e000c560 tx=0x7f24e0007d30 comp rx=0 tx=0).stop 2023-09-30T00:53:14.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 1 -- 172.21.15.40:0/2593945652 shutdown_connections 2023-09-30T00:53:14.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 1 --2- 172.21.15.40:0/2593945652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24ec13ac20 0x7f24ec1510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 1 --2- 172.21.15.40:0/2593945652 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f24ec0a5fb0 0x7f24ec13e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 1 --2- 172.21.15.40:0/2593945652 >> v2:172.21.15.135:6800/116426 conn(0x7f24d8068960 0x7f24d806ae10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 1 --2- 172.21.15.40:0/2593945652 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f24ec0b1460 0x7f24ec0a79f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 1 -- 172.21.15.40:0/2593945652 >> 172.21.15.40:0/2593945652 conn(0x7f24ec0631f0 msgr2=0x7f24ec144aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:14.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 1 -- 172.21.15.40:0/2593945652 shutdown_connections 2023-09-30T00:53:14.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 1 -- 172.21.15.40:0/2593945652 wait complete. 2023-09-30T00:53:14.592 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.590+0000 7f24f28f6700 1 librados: shutdown 2023-09-30T00:53:14.593 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 4 to deep-scrub 2023-09-30T00:53:14.603 INFO:tasks.ceph:Scrubbing osd.5 2023-09-30T00:53:14.604 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:53:14.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.741+0000 7f3b8e5f8700 10 monclient: get_monmap_and_config 2023-09-30T00:53:14.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.741+0000 7f3b8e5f8700 10 monclient: build_initial_monmap 2023-09-30T00:53:14.742 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.741+0000 7f3b8e5f8700 10 monclient: monmap: 2023-09-30T00:53:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:14.742420+0000 2023-09-30T00:53:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:14.742420+0000 2023-09-30T00:53:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:14.743 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:14.743 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 1 Processor -- start 2023-09-30T00:53:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 1 -- start start 2023-09-30T00:53:14.744 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 10 monclient: init 2023-09-30T00:53:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-09-30T00:53:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3b880a5710 0x7f3b880b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 10 monclient(hunting): picked mon.noname-a con 0x7f3b880a5710 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b880a7630 0x7f3b880a7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 10 monclient(hunting): picked mon.noname-b con 0x7f3b880a7630 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f3b880a7fe0 0x7f3b880af3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 10 monclient(hunting): picked mon.noname-c con 0x7f3b880a7fe0 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b880b1770 con 0x7f3b880a5710 2023-09-30T00:53:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:14.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b880a5ae0 con 0x7f3b880a7fe0 2023-09-30T00:53:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b880b1040 con 0x7f3b880a7630 2023-09-30T00:53:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.742+0000 7f3b8e5f8700 10 monclient(hunting): authenticate will time out at 5656.043896s 2023-09-30T00:53:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.743+0000 7f3b877fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b880a7630 0x7f3b880a7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.743+0000 7f3b87fff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3b880a5710 0x7f3b880b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.743+0000 7f3b877fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b880a7630 0x7f3b880a7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:37560/0 (socket says 172.21.15.40:37560) 2023-09-30T00:53:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.743+0000 7f3b877fe700 1 -- 172.21.15.40:0/2188434710 learned_addr learned my addr 172.21.15.40:0/2188434710 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.743+0000 7f3b877fe700 10 monclient(hunting): get_auth_request con 0x7f3b880a7630 auth_method 0 2023-09-30T00:53:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.743+0000 7f3b877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.743+0000 7f3b877fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:14.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.743+0000 7f3b877fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.743+0000 7f3b8cb95700 1 --2- 172.21.15.40:0/2188434710 >> v2:172.21.15.135:3300/0 conn(0x7f3b880a7fe0 0x7f3b880af3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b87fff700 10 monclient(hunting): get_auth_request con 0x7f3b880a5710 auth_method 0 2023-09-30T00:53:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b87fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b87fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b8cb95700 10 monclient(hunting): get_auth_request con 0x7f3b880a7fe0 auth_method 0 2023-09-30T00:53:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b8cb95700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b8cb95700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b8cb95700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:14.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:14.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:14.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:14.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:14.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b8cb95700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:14.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b8cb95700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:14.749 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.744+0000 7f3b8cb95700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:14.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.746+0000 7f3b877fe700 10 monclient(hunting): handle_auth_done global_id 4739 payload 274 2023-09-30T00:53:14.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.746+0000 7f3b877fe700 1 -- 172.21.15.40:0/2188434710 >> v2:172.21.15.135:3300/0 conn(0x7f3b880a7fe0 msgr2=0x7f3b880af3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:14.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.746+0000 7f3b877fe700 1 --2- 172.21.15.40:0/2188434710 >> v2:172.21.15.135:3300/0 conn(0x7f3b880a7fe0 0x7f3b880af3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.746+0000 7f3b877fe700 1 -- 172.21.15.40:0/2188434710 >> v2:172.21.15.40:3300/0 conn(0x7f3b880a5710 msgr2=0x7f3b880b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:14.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.746+0000 7f3b877fe700 1 --2- 172.21.15.40:0/2188434710 >> v2:172.21.15.40:3300/0 conn(0x7f3b880a5710 0x7f3b880b0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.746+0000 7f3b877fe700 10 monclient: _finish_hunting 0 2023-09-30T00:53:14.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.746+0000 7f3b877fe700 1 monclient: found mon.noname-b 2023-09-30T00:53:14.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.746+0000 7f3b877fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:14.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.746+0000 7f3b877fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:53:14.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.746+0000 7f3b877fe700 1 -- 172.21.15.40:0/2188434710 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b880b1440 con 0x7f3b880a7630 2023-09-30T00:53:14.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.749+0000 7f3b877fe700 1 --2- 172.21.15.40:0/2188434710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b880a7630 0x7f3b880a7aa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3b880abac0 tx=0x7f3b7c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fdfd05fde57e8f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:14.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b86ffd700 1 -- 172.21.15.40:0/2188434710 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c00b640 con 0x7f3b880a7630 2023-09-30T00:53:14.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:14.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b86ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:14.754 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b86ffd700 10 monclient: dump: 2023-09-30T00:53:14.754 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:14.754 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:14.754 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.755 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.755 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:14.755 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:14.755 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:14.755 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:14.755 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:14.755 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:14.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b86ffd700 10 monclient: _finish_auth 0 2023-09-30T00:53:14.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b86ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:14.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b86ffd700 1 -- 172.21.15.40:0/2188434710 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7c00b7a0 con 0x7f3b880a7630 2023-09-30T00:53:14.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b8e5f8700 5 monclient: authenticate success, global_id 4739 2023-09-30T00:53:14.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b8e5f8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:14.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b86ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:14.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b8e5f8700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:14.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b86ffd700 1 -- 172.21.15.40:0/2188434710 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c00bb60 con 0x7f3b880a7630 2023-09-30T00:53:14.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b8e5f8700 4 set_mon_vals no callback set 2023-09-30T00:53:14.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:14.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b86ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:53:14.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.753+0000 7f3b86ffd700 10 monclient: dump: 2023-09-30T00:53:14.756 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:14.756 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:14.756 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.756 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.756 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:14.756 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:14.757 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:14.757 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:14.757 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:14.757 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:14.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.754+0000 7f3b8e5f8700 10 monclient: shutdown 2023-09-30T00:53:14.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.754+0000 7f3b8e5f8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:14.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.754+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/2188434710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b880a7630 msgr2=0x7f3b880a7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:14.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.754+0000 7f3b8e5f8700 1 --2- 172.21.15.40:0/2188434710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b880a7630 0x7f3b880a7aa0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3b880abac0 tx=0x7f3b7c009250 comp rx=0 tx=0).stop 2023-09-30T00:53:14.760 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.754+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/2188434710 shutdown_connections 2023-09-30T00:53:14.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.754+0000 7f3b8e5f8700 1 --2- 172.21.15.40:0/2188434710 >> v2:172.21.15.135:3300/0 conn(0x7f3b880a7fe0 0x7f3b880af3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.754+0000 7f3b8e5f8700 1 --2- 172.21.15.40:0/2188434710 >> v2:172.21.15.40:3300/0 conn(0x7f3b880a5710 0x7f3b880b0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.754+0000 7f3b8e5f8700 1 --2- 172.21.15.40:0/2188434710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b880a7630 0x7f3b880a7aa0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.754+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/2188434710 >> 172.21.15.40:0/2188434710 conn(0x7f3b88069060 msgr2=0x7f3b880631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:14.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.754+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/2188434710 shutdown_connections 2023-09-30T00:53:14.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.754+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/2188434710 wait complete. 2023-09-30T00:53:14.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.755+0000 7f3b8e5f8700 10 monclient: build_initial_monmap 2023-09-30T00:53:14.761 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.755+0000 7f3b8e5f8700 10 monclient: monmap: 2023-09-30T00:53:14.761 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:14.761 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:14.761 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:14.761 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:14.761 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:14.762 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:14.762 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:14.762 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:14.762 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:14.762 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:14.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.755+0000 7f3b8e5f8700 1 librados: starting msgr at 2023-09-30T00:53:14.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.755+0000 7f3b8e5f8700 1 librados: starting objecter 2023-09-30T00:53:14.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.755+0000 7f3b8e5f8700 1 Processor -- start 2023-09-30T00:53:14.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 1 -- start start 2023-09-30T00:53:14.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 1 librados: setting wanted keys 2023-09-30T00:53:14.762 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 1 librados: calling monclient init 2023-09-30T00:53:14.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 10 monclient: init 2023-09-30T00:53:14.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:14.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:53:14.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b880a5710 0x7f3b880ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:14.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 10 monclient(hunting): picked mon.noname-a con 0x7f3b880a5710 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:14.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b880a7630 0x7f3b8813dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:14.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 10 monclient(hunting): picked mon.noname-b con 0x7f3b880a7630 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:14.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b880a7fe0 0x7f3b880aa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:14.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 10 monclient(hunting): picked mon.noname-c con 0x7f3b880a7fe0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:14.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:14.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8813e380 con 0x7f3b880a5710 2023-09-30T00:53:14.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:14.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8813cd50 con 0x7f3b880a7630 2023-09-30T00:53:14.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:14.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8813f2b0 con 0x7f3b880a7fe0 2023-09-30T00:53:14.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:14.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.756+0000 7f3b8e5f8700 10 monclient(hunting): authenticate will time out at 5656.057893s 2023-09-30T00:53:14.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.757+0000 7f3b87fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b880a5710 0x7f3b880ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:14.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.757+0000 7f3b877fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b880a7630 0x7f3b8813dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:14.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.757+0000 7f3b87fff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b880a5710 0x7f3b880ae270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46154/0 (socket says 172.21.15.40:46154) 2023-09-30T00:53:14.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.757+0000 7f3b87fff700 1 -- 172.21.15.40:0/1280524924 learned_addr learned my addr 172.21.15.40:0/1280524924 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:14.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.757+0000 7f3b87fff700 10 monclient(hunting): get_auth_request con 0x7f3b880a5710 auth_method 0 2023-09-30T00:53:14.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.757+0000 7f3b87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:14.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.757+0000 7f3b87fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:14.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.757+0000 7f3b87fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.757+0000 7f3b8cb95700 1 --2- 172.21.15.40:0/1280524924 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b880a7fe0 0x7f3b880aa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.758+0000 7f3b877fe700 10 monclient(hunting): get_auth_request con 0x7f3b880a7630 auth_method 0 2023-09-30T00:53:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.758+0000 7f3b877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.758+0000 7f3b877fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.758+0000 7f3b877fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.758+0000 7f3b8cb95700 10 monclient(hunting): get_auth_request con 0x7f3b880a7fe0 auth_method 0 2023-09-30T00:53:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.758+0000 7f3b8cb95700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.758+0000 7f3b8cb95700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.758+0000 7f3b8cb95700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.758+0000 7f3b877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.758+0000 7f3b877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.758+0000 7f3b877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.758+0000 7f3b87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:14.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.758+0000 7f3b87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:14.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.758+0000 7f3b87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:14.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.759+0000 7f3b8cb95700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:14.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.759+0000 7f3b8cb95700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:14.766 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.759+0000 7f3b8cb95700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:14.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.761+0000 7f3b877fe700 10 monclient(hunting): handle_auth_done global_id 4960 payload 824 2023-09-30T00:53:14.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.761+0000 7f3b877fe700 1 -- 172.21.15.40:0/1280524924 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b880a7fe0 msgr2=0x7f3b880aa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:14.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.761+0000 7f3b877fe700 1 --2- 172.21.15.40:0/1280524924 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f3b880a7fe0 0x7f3b880aa2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.761+0000 7f3b877fe700 1 -- 172.21.15.40:0/1280524924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b880a5710 msgr2=0x7f3b880ae270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:14.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.761+0000 7f3b877fe700 1 --2- 172.21.15.40:0/1280524924 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3b880a5710 0x7f3b880ae270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.761+0000 7f3b877fe700 10 monclient: _finish_hunting 0 2023-09-30T00:53:14.767 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.761+0000 7f3b877fe700 1 monclient: found mon.noname-b 2023-09-30T00:53:14.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.761+0000 7f3b877fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:14.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.761+0000 7f3b877fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:14.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.761+0000 7f3b877fe700 1 -- 172.21.15.40:0/1280524924 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7c0094f0 con 0x7f3b880a7630 2023-09-30T00:53:14.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.763+0000 7f3b877fe700 1 --2- 172.21.15.40:0/1280524924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b880a7630 0x7f3b8813dc90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3b88069b00 tx=0x7f3b7c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:14.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.763+0000 7f3b6ffff700 10 client.?.objecter ms_handle_connect 0x7f3b880a7630 2023-09-30T00:53:14.768 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.763+0000 7f3b6ffff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:14.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.767+0000 7f3b6ffff700 1 -- 172.21.15.40:0/1280524924 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c00b910 con 0x7f3b880a7630 2023-09-30T00:53:14.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.767+0000 7f3b6ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:14.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.767+0000 7f3b6ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:14.770 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.767+0000 7f3b6ffff700 10 monclient: dump: 2023-09-30T00:53:14.770 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:14.770 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:14.770 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.771 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.771 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:14.771 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:14.771 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:14.771 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:14.771 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:14.771 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:14.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b6ffff700 10 monclient: _finish_auth 0 2023-09-30T00:53:14.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b6ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:14.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b6ffff700 1 -- 172.21.15.40:0/1280524924 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7c00ba90 con 0x7f3b880a7630 2023-09-30T00:53:14.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 5 monclient: authenticate success, global_id 4960 2023-09-30T00:53:14.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 10 monclient: _renew_subs 2023-09-30T00:53:14.771 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:14.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/1280524924 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b8813cfc0 con 0x7f3b880a7630 2023-09-30T00:53:14.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b6ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:14.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b6ffff700 1 -- 172.21.15.40:0/1280524924 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c011670 con 0x7f3b880a7630 2023-09-30T00:53:14.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b6ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:14.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b6ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:14.772 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b6ffff700 10 monclient: dump: 2023-09-30T00:53:14.772 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:14.772 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:14.772 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.772 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:14.772 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:14.772 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:14.773 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:14.773 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:14.773 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:14.773 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:14.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b867fc700 4 set_mon_vals no callback set 2023-09-30T00:53:14.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 10 client.4960.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:14.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 10 monclient: _renew_subs 2023-09-30T00:53:14.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:14.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/1280524924 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b880aa9c0 con 0x7f3b880a7630 2023-09-30T00:53:14.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 1 librados: init done 2023-09-30T00:53:14.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 10 client.4960.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:14.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 15 client.4960.objecter _session_command_op_assign -1 1 2023-09-30T00:53:14.773 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 20 client.4960.objecter _get_session osd=-1 returning homeless 2023-09-30T00:53:14.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 10 client.4960.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:14.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:14.774 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.768+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/1280524924 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3b88064a00 con 0x7f3b880a7630 2023-09-30T00:53:14.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.789+0000 7f3b6ffff700 1 -- 172.21.15.40:0/1280524924 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3b7c011a20 con 0x7f3b880a7630 2023-09-30T00:53:14.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.789+0000 7f3b6ffff700 1 --2- 172.21.15.40:0/1280524924 >> v2:172.21.15.135:6800/116426 conn(0x7f3b74068890 0x7f3b7406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:14.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.790+0000 7f3b87fff700 1 --2- 172.21.15.40:0/1280524924 >> v2:172.21.15.135:6800/116426 conn(0x7f3b74068890 0x7f3b7406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:14.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.790+0000 7f3b87fff700 10 monclient: get_auth_request con 0x7f3b74068890 auth_method 0 2023-09-30T00:53:14.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.790+0000 7f3b6ffff700 10 client.4960.objecter ms_handle_connect 0x7f3b74068890 2023-09-30T00:53:14.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.790+0000 7f3b87fff700 1 --2- 172.21.15.40:0/1280524924 >> v2:172.21.15.135:6800/116426 conn(0x7f3b74068890 0x7f3b7406ad40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009dd0 tx=0x7f3b780091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:14.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 1 -- 172.21.15.40:0/1280524924 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3b7c057830 con 0x7f3b880a7630 2023-09-30T00:53:14.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 10 client.4960.objecter ms_dispatch 0x7f3b88001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:14.795 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 3 client.4960.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:14.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 3 client.4960.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:14.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 10 client.4960.objecter checking command 1 2023-09-30T00:53:14.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 1 --2- 172.21.15.40:0/1280524924 >> v2:172.21.15.135:6801/117130 conn(0x7f3b7406e800 0x7f3b74070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:14.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 20 client.4960.objecter _get_session s=0x7f3b7406e150 osd=5 3 2023-09-30T00:53:14.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 20 client.4960.objecter put_session s=0x7f3b7406e150 osd=5 3 2023-09-30T00:53:14.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 15 client.4960.objecter _session_command_op_remove -1 1 2023-09-30T00:53:14.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 20 client.4960.objecter _get_session s=0x7f3b7406e150 osd=5 3 2023-09-30T00:53:14.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 20 client.4960.objecter get_session s=0x7f3b7406e150 osd=5 3 2023-09-30T00:53:14.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 15 client.4960.objecter _session_command_op_assign 5 1 2023-09-30T00:53:14.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 20 client.4960.objecter put_session s=0x7f3b7406e150 osd=5 4 2023-09-30T00:53:14.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 10 client.4960.objecter _send_command 1 2023-09-30T00:53:14.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 1 -- 172.21.15.40:0/1280524924 --> v2:172.21.15.135:6801/117130 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b74071280 con 0x7f3b7406e800 2023-09-30T00:53:14.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.795+0000 7f3b6ffff700 20 client.4960.objecter dump_active .. 0 homeless 2023-09-30T00:53:14.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.796+0000 7f3b8cb95700 1 --2- 172.21.15.40:0/1280524924 >> v2:172.21.15.135:6801/117130 conn(0x7f3b7406e800 0x7f3b74070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:14.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.796+0000 7f3b8cb95700 10 monclient: get_auth_request con 0x7f3b7406e800 auth_method 0 2023-09-30T00:53:14.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.796+0000 7f3b6ffff700 1 -- 172.21.15.40:0/1280524924 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b7c053470 con 0x7f3b880a7630 2023-09-30T00:53:14.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.796+0000 7f3b6ffff700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:53:14.800 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.800+0000 7f3b8cb95700 1 --2- 172.21.15.40:0/1280524924 >> v2:172.21.15.135:6801/117130 conn(0x7f3b7406e800 0x7f3b74070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:14.801 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.800+0000 7f3b6ffff700 10 client.4960.objecter ms_handle_connect 0x7f3b7406e800 2023-09-30T00:53:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.824+0000 7f3b6ffff700 1 -- 172.21.15.40:0/1280524924 <== osd.5 v2:172.21.15.135:6801/117130 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f3b74071280 con 0x7f3b7406e800 2023-09-30T00:53:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.824+0000 7f3b6ffff700 10 client.4960.objecter ms_dispatch 0x7f3b88001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:53:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.824+0000 7f3b6ffff700 10 client.4960.objecter _finish_command 1 = system:0 2023-09-30T00:53:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.824+0000 7f3b6ffff700 20 client.4960.objecter put_session s=0x7f3b7406e150 osd=5 4 2023-09-30T00:53:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.824+0000 7f3b6ffff700 15 client.4960.objecter _session_command_op_remove 5 1 2023-09-30T00:53:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.838+0000 7f3b8e5f8700 10 client.4960.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T00:53:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.838+0000 7f3b8e5f8700 15 client.4960.objecter _session_command_op_assign -1 2 2023-09-30T00:53:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.838+0000 7f3b8e5f8700 20 client.4960.objecter _get_session s=0x7f3b7406e150 osd=5 3 2023-09-30T00:53:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.838+0000 7f3b8e5f8700 20 client.4960.objecter put_session s=0x7f3b7406e150 osd=5 3 2023-09-30T00:53:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.838+0000 7f3b8e5f8700 20 client.4960.objecter _get_session s=0x7f3b7406e150 osd=5 3 2023-09-30T00:53:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.838+0000 7f3b8e5f8700 15 client.4960.objecter _session_command_op_remove -1 2 2023-09-30T00:53:14.839 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.838+0000 7f3b8e5f8700 20 client.4960.objecter get_session s=0x7f3b7406e150 osd=5 3 2023-09-30T00:53:14.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.838+0000 7f3b8e5f8700 15 client.4960.objecter _session_command_op_assign 5 2 2023-09-30T00:53:14.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.838+0000 7f3b8e5f8700 20 client.4960.objecter put_session s=0x7f3b7406e150 osd=5 4 2023-09-30T00:53:14.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.838+0000 7f3b8e5f8700 10 client.4960.objecter _send_command 2 2023-09-30T00:53:14.840 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.838+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/1280524924 --> v2:172.21.15.135:6801/117130 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3b880a7210 con 0x7f3b7406e800 2023-09-30T00:53:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.861+0000 7f3b6ffff700 1 -- 172.21.15.40:0/1280524924 <== osd.5 v2:172.21.15.135:6801/117130 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f3b880a7210 con 0x7f3b7406e800 2023-09-30T00:53:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.861+0000 7f3b6ffff700 10 client.4960.objecter ms_dispatch 0x7f3b88001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:53:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.861+0000 7f3b6ffff700 10 client.4960.objecter _finish_command 2 = system:0 2023-09-30T00:53:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.861+0000 7f3b6ffff700 20 client.4960.objecter put_session s=0x7f3b7406e150 osd=5 4 2023-09-30T00:53:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.861+0000 7f3b6ffff700 15 client.4960.objecter _session_command_op_remove 5 2 2023-09-30T00:53:14.862 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-09-30T00:53:14.862 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:53:14.862 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-09-30T00:53:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.861+0000 7f3b8e5f8700 10 librados: watch_flush enter 2023-09-30T00:53:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.861+0000 7f3b8e5f8700 10 librados: watch_flush exit 2023-09-30T00:53:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.861+0000 7f3b8e5f8700 10 client.4960.objecter close_session for osd.5 2023-09-30T00:53:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.861+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/1280524924 >> v2:172.21.15.135:6801/117130 conn(0x7f3b7406e800 msgr2=0x7f3b74070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.861+0000 7f3b8e5f8700 1 --2- 172.21.15.40:0/1280524924 >> v2:172.21.15.135:6801/117130 conn(0x7f3b7406e800 0x7f3b74070c10 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.861+0000 7f3b8e5f8700 20 client.4960.objecter put_session s=0x7f3b7406e150 osd=5 1 2023-09-30T00:53:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.861+0000 7f3b8e5f8700 20 client.4960.objecter shutdown clearing up homeless session... 2023-09-30T00:53:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.861+0000 7f3b8e5f8700 10 client.4960.objecter successfully canceled tick 2023-09-30T00:53:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.862+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/1280524924 >> v2:172.21.15.135:6800/116426 conn(0x7f3b74068890 msgr2=0x7f3b7406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.862+0000 7f3b8e5f8700 1 --2- 172.21.15.40:0/1280524924 >> v2:172.21.15.135:6800/116426 conn(0x7f3b74068890 0x7f3b7406ad40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009dd0 tx=0x7f3b780091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.862+0000 7f3b8e5f8700 10 monclient: shutdown 2023-09-30T00:53:14.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.862+0000 7f3b8e5f8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:14.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.862+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/1280524924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b880a7630 msgr2=0x7f3b8813dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:14.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.862+0000 7f3b8e5f8700 1 --2- 172.21.15.40:0/1280524924 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f3b880a7630 0x7f3b8813dc90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3b88069b00 tx=0x7f3b7c017000 comp rx=0 tx=0).stop 2023-09-30T00:53:14.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.862+0000 7f3b8cb95700 1 -- 172.21.15.40:0/1280524924 reap_dead start 2023-09-30T00:53:14.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.862+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/1280524924 shutdown_connections 2023-09-30T00:53:14.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.862+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/1280524924 >> 172.21.15.40:0/1280524924 conn(0x7f3b88069060 msgr2=0x7f3b880b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:14.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.862+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/1280524924 shutdown_connections 2023-09-30T00:53:14.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.862+0000 7f3b8e5f8700 1 -- 172.21.15.40:0/1280524924 wait complete. 2023-09-30T00:53:14.865 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:14.862+0000 7f3b8e5f8700 1 librados: shutdown 2023-09-30T00:53:14.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-09-30T00:53:15.015 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.015+0000 7f4a993ff700 10 monclient: get_monmap_and_config 2023-09-30T00:53:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.015+0000 7f4a993ff700 10 monclient: build_initial_monmap 2023-09-30T00:53:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.015+0000 7f4a993ff700 10 monclient: monmap: 2023-09-30T00:53:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:15.016647+0000 2023-09-30T00:53:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:15.016647+0000 2023-09-30T00:53:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:15.016 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:15.017 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:15.017 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 1 Processor -- start 2023-09-30T00:53:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 1 -- start start 2023-09-30T00:53:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 10 monclient: init 2023-09-30T00:53:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2023-09-30T00:53:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a940b1460 0x7f4a940b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 10 monclient(hunting): picked mon.noname-a con 0x7f4a940b1460 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a940a5fb0 0x7f4a94141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 10 monclient(hunting): picked mon.noname-b con 0x7f4a940a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a940af410 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:15.019 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a940a71f0 con 0x7f4a940af410 2023-09-30T00:53:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a940a74d0 con 0x7f4a940b1460 2023-09-30T00:53:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a940a63c0 con 0x7f4a940a5fb0 2023-09-30T00:53:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.016+0000 7f4a993ff700 10 monclient(hunting): authenticate will time out at 5656.317981s 2023-09-30T00:53:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.017+0000 7f4a92ffd700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4a940b1460 0x7f4a940b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.017+0000 7f4a927fc700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a940a5fb0 0x7f4a94141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.017+0000 7f4a937fe700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f4a940af410 0x7f4a940af890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:33234/0 (socket says 172.21.15.40:33234) 2023-09-30T00:53:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.017+0000 7f4a937fe700 1 -- 172.21.15.40:0/2099053857 learned_addr learned my addr 172.21.15.40:0/2099053857 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:53:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.018+0000 7f4a92ffd700 10 monclient(hunting): get_auth_request con 0x7f4a940b1460 auth_method 0 2023-09-30T00:53:15.020 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.018+0000 7f4a92ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.018+0000 7f4a92ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.018+0000 7f4a92ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.018+0000 7f4a927fc700 10 monclient(hunting): get_auth_request con 0x7f4a940a5fb0 auth_method 0 2023-09-30T00:53:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.018+0000 7f4a927fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.018+0000 7f4a927fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.021 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.018+0000 7f4a927fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.019+0000 7f4a92ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.019+0000 7f4a92ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.019+0000 7f4a92ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.019+0000 7f4a927fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.019+0000 7f4a927fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.019+0000 7f4a927fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.020+0000 7f4a91ffb700 1 -- 172.21.15.40:0/2099053857 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910423065 0 0) 0x7f4a940a71f0 con 0x7f4a940af410 2023-09-30T00:53:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.020+0000 7f4a91ffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.020+0000 7f4a91ffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.022 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.020+0000 7f4a91ffb700 10 monclient(hunting): my global_id is 4963 2023-09-30T00:53:15.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.020+0000 7f4a91ffb700 1 -- 172.21.15.40:0/2099053857 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7c003570 con 0x7f4a940af410 2023-09-30T00:53:15.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.022+0000 7f4a92ffd700 10 monclient(hunting): handle_auth_done global_id 4881 payload 274 2023-09-30T00:53:15.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.022+0000 7f4a92ffd700 1 -- 172.21.15.40:0/2099053857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a940a5fb0 msgr2=0x7f4a94141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:15.023 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.022+0000 7f4a92ffd700 1 --2- 172.21.15.40:0/2099053857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a940a5fb0 0x7f4a94141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.022+0000 7f4a92ffd700 1 -- 172.21.15.40:0/2099053857 >> v1:172.21.15.135:6789/0 conn(0x7f4a940af410 legacy=0x7f4a940af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.022+0000 7f4a92ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:53:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.022+0000 7f4a92ffd700 1 monclient: found mon.noname-a 2023-09-30T00:53:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.022+0000 7f4a92ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.022+0000 7f4a92ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:53:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.022+0000 7f4a92ffd700 1 -- 172.21.15.40:0/2099053857 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a940a56d0 con 0x7f4a940b1460 2023-09-30T00:53:15.024 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.022+0000 7f4a927fc700 1 --2- 172.21.15.40:0/2099053857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a940a5fb0 0x7f4a94141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:53:15.025 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.025+0000 7f4a92ffd700 1 --2- 172.21.15.40:0/2099053857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a940b1460 0x7f4a940b1830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4a840098e0 tx=0x7f4a840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d40a3f168954e26 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:15.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.027+0000 7f4a91ffb700 1 -- 172.21.15.40:0/2099053857 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a84011630 con 0x7f4a940b1460 2023-09-30T00:53:15.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.027+0000 7f4a91ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:15.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.027+0000 7f4a91ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:53:15.028 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.027+0000 7f4a91ffb700 10 monclient: dump: 2023-09-30T00:53:15.028 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:15.029 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:15.029 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.029 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.029 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:15.029 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:15.029 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:15.029 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:15.029 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:15.029 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.028+0000 7f4a91ffb700 10 monclient: _finish_auth 0 2023-09-30T00:53:15.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.028+0000 7f4a91ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:15.029 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.028+0000 7f4a993ff700 5 monclient: authenticate success, global_id 4881 2023-09-30T00:53:15.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.028+0000 7f4a993ff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:15.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.030+0000 7f4a91ffb700 1 -- 172.21.15.40:0/2099053857 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a84011790 con 0x7f4a940b1460 2023-09-30T00:53:15.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.030+0000 7f4a91ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:15.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.030+0000 7f4a91ffb700 1 -- 172.21.15.40:0/2099053857 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a84011b10 con 0x7f4a940b1460 2023-09-30T00:53:15.030 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.030+0000 7f4a91ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.030+0000 7f4a91ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:53:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.030+0000 7f4a91ffb700 10 monclient: dump: 2023-09-30T00:53:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:15.031 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:15.031 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.030+0000 7f4a993ff700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.030+0000 7f4a993ff700 4 set_mon_vals no callback set 2023-09-30T00:53:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.030+0000 7f4a993ff700 10 monclient: shutdown 2023-09-30T00:53:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.030+0000 7f4a993ff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.030+0000 7f4a993ff700 1 -- 172.21.15.40:0/2099053857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a940b1460 msgr2=0x7f4a940b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.030+0000 7f4a993ff700 1 --2- 172.21.15.40:0/2099053857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a940b1460 0x7f4a940b1830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4a840098e0 tx=0x7f4a840091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.031+0000 7f4a993ff700 1 -- 172.21.15.40:0/2099053857 shutdown_connections 2023-09-30T00:53:15.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.031+0000 7f4a993ff700 1 --2- 172.21.15.40:0/2099053857 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a940b1460 0x7f4a940b1830 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.031+0000 7f4a993ff700 1 --2- 172.21.15.40:0/2099053857 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a940a5fb0 0x7f4a94141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.031+0000 7f4a993ff700 1 -- 172.21.15.40:0/2099053857 >> 172.21.15.40:0/2099053857 conn(0x7f4a940631f0 msgr2=0x7f4a940635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.031+0000 7f4a993ff700 1 -- 172.21.15.40:0/2099053857 shutdown_connections 2023-09-30T00:53:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.031+0000 7f4a993ff700 1 -- 172.21.15.40:0/2099053857 wait complete. 2023-09-30T00:53:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.031+0000 7f4a993ff700 10 monclient: build_initial_monmap 2023-09-30T00:53:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.031+0000 7f4a993ff700 10 monclient: monmap: 2023-09-30T00:53:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:15.035 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:15.036 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.032+0000 7f4a993ff700 1 librados: starting msgr at 2023-09-30T00:53:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.032+0000 7f4a993ff700 1 librados: starting objecter 2023-09-30T00:53:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.032+0000 7f4a993ff700 1 Processor -- start 2023-09-30T00:53:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.032+0000 7f4a993ff700 1 -- start start 2023-09-30T00:53:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.032+0000 7f4a993ff700 1 librados: setting wanted keys 2023-09-30T00:53:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.032+0000 7f4a993ff700 1 librados: calling monclient init 2023-09-30T00:53:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.032+0000 7f4a993ff700 10 monclient: init 2023-09-30T00:53:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.032+0000 7f4a993ff700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:53:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a940a5fb0 0x7f4a94151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 10 monclient(hunting): picked mon.noname-c con 0x7f4a940a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a940af410 0x7f4a94133dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 10 monclient(hunting): picked mon.noname-b con 0x7f4a940af410 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a940b1460 0x7f4a94134300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 10 monclient(hunting): picked mon.noname-a con 0x7f4a940b1460 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9413d5e0 con 0x7f4a940b1460 2023-09-30T00:53:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9413c380 con 0x7f4a940af410 2023-09-30T00:53:15.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9413f8a0 con 0x7f4a940a5fb0 2023-09-30T00:53:15.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:15.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a993ff700 10 monclient(hunting): authenticate will time out at 5656.334110s 2023-09-30T00:53:15.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a92ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a940a5fb0 0x7f4a94151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.033+0000 7f4a927fc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a940af410 0x7f4a94133dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a92ffd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a940a5fb0 0x7f4a94151220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:37588/0 (socket says 172.21.15.40:37588) 2023-09-30T00:53:15.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a92ffd700 1 -- 172.21.15.40:0/884766622 learned_addr learned my addr 172.21.15.40:0/884766622 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:15.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a92ffd700 10 monclient(hunting): get_auth_request con 0x7f4a940a5fb0 auth_method 0 2023-09-30T00:53:15.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a92ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:15.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a92ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a92ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a927fc700 10 monclient(hunting): get_auth_request con 0x7f4a940af410 auth_method 0 2023-09-30T00:53:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a927fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a927fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a927fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a937fe700 1 --2- 172.21.15.40:0/884766622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a940b1460 0x7f4a94134300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a92ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a92ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a92ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a927fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:15.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a927fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.034+0000 7f4a927fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.035+0000 7f4a937fe700 10 monclient(hunting): get_auth_request con 0x7f4a940b1460 auth_method 0 2023-09-30T00:53:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.035+0000 7f4a937fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.035+0000 7f4a937fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.035+0000 7f4a937fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.035+0000 7f4a937fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.035+0000 7f4a937fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.035+0000 7f4a937fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.037+0000 7f4a927fc700 10 monclient(hunting): handle_auth_done global_id 4966 payload 824 2023-09-30T00:53:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.037+0000 7f4a927fc700 1 -- 172.21.15.40:0/884766622 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a940a5fb0 msgr2=0x7f4a94151220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.037+0000 7f4a927fc700 1 --2- 172.21.15.40:0/884766622 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a940a5fb0 0x7f4a94151220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.037+0000 7f4a927fc700 1 -- 172.21.15.40:0/884766622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a940b1460 msgr2=0x7f4a94134300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.037+0000 7f4a927fc700 1 --2- 172.21.15.40:0/884766622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a940b1460 0x7f4a94134300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.037+0000 7f4a927fc700 10 monclient: _finish_hunting 0 2023-09-30T00:53:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.037+0000 7f4a927fc700 1 monclient: found mon.noname-b 2023-09-30T00:53:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.037+0000 7f4a927fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.037+0000 7f4a927fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.037+0000 7f4a927fc700 1 -- 172.21.15.40:0/884766622 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a88009420 con 0x7f4a940af410 2023-09-30T00:53:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.040+0000 7f4a927fc700 1 --2- 172.21.15.40:0/884766622 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a940af410 0x7f4a94133dc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800ec80 tx=0x7f4a880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.040+0000 7f4a7affd700 10 client.?.objecter ms_handle_connect 0x7f4a940af410 2023-09-30T00:53:15.042 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.040+0000 7f4a7affd700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.043+0000 7f4a7affd700 1 -- 172.21.15.40:0/884766622 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8800c850 con 0x7f4a940af410 2023-09-30T00:53:15.044 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a7affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a7affd700 10 monclient: dump: 2023-09-30T00:53:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:15.045 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a7affd700 10 monclient: _finish_auth 0 2023-09-30T00:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a7affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a7affd700 1 -- 172.21.15.40:0/884766622 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8800c9b0 con 0x7f4a940af410 2023-09-30T00:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a7affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a7affd700 1 -- 172.21.15.40:0/884766622 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8800cd70 con 0x7f4a940af410 2023-09-30T00:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a993ff700 5 monclient: authenticate success, global_id 4966 2023-09-30T00:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a993ff700 10 monclient: _renew_subs 2023-09-30T00:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a993ff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a993ff700 1 -- 172.21.15.40:0/884766622 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a84009420 con 0x7f4a940af410 2023-09-30T00:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:15.046 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a7affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a7affd700 10 monclient: dump: 2023-09-30T00:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a917fa700 4 set_mon_vals no callback set 2023-09-30T00:53:15.047 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a993ff700 10 client.4966.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a993ff700 10 monclient: _renew_subs 2023-09-30T00:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a993ff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a993ff700 1 -- 172.21.15.40:0/884766622 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a941404e0 con 0x7f4a940af410 2023-09-30T00:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.044+0000 7f4a993ff700 1 librados: init done 2023-09-30T00:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.045+0000 7f4a993ff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.045+0000 7f4a993ff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:15.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.045+0000 7f4a993ff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:15.049 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.045+0000 7f4a993ff700 1 -- 172.21.15.40:0/884766622 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a941412c0 con 0x7f4a940af410 2023-09-30T00:53:15.113 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.113+0000 7f4a7affd700 1 -- 172.21.15.40:0/884766622 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f4a88013680 con 0x7f4a940af410 2023-09-30T00:53:15.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.113+0000 7f4a7affd700 1 --2- 172.21.15.40:0/884766622 >> v2:172.21.15.135:6800/116426 conn(0x7f4a80068960 0x7f4a8006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.114+0000 7f4a92ffd700 1 --2- 172.21.15.40:0/884766622 >> v2:172.21.15.135:6800/116426 conn(0x7f4a80068960 0x7f4a8006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.114+0000 7f4a92ffd700 10 monclient: get_auth_request con 0x7f4a80068960 auth_method 0 2023-09-30T00:53:15.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.114+0000 7f4a92ffd700 1 --2- 172.21.15.40:0/884766622 >> v2:172.21.15.135:6800/116426 conn(0x7f4a80068960 0x7f4a8006ae10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4a840091e0 tx=0x7f4a84017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:15.115 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.114+0000 7f4a7affd700 10 client.4966.objecter ms_handle_connect 0x7f4a80068960 2023-09-30T00:53:15.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.295+0000 7f4a7affd700 1 -- 172.21.15.40:0/884766622 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f4a880563b0 con 0x7f4a940af410 2023-09-30T00:53:15.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.296+0000 7f4a7affd700 10 client.4966.objecter ms_dispatch 0x7f4a94001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:15.296 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.296+0000 7f4a7affd700 3 client.4966.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:15.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.296+0000 7f4a7affd700 3 client.4966.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:15.297 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.296+0000 7f4a7affd700 20 client.4966.objecter dump_active .. 0 homeless 2023-09-30T00:53:15.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.308+0000 7f4a7affd700 1 -- 172.21.15.40:0/884766622 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4a88051ff0 con 0x7f4a940af410 2023-09-30T00:53:15.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.308+0000 7f4a7affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:15.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.308+0000 7f4a7affd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:15.547 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.547+0000 7f4a993ff700 1 -- 172.21.15.40:0/884766622 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a941412c0 con 0x7f4a80068960 2023-09-30T00:53:15.548 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.548+0000 7f4a7affd700 1 -- 172.21.15.40:0/884766622 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4a941412c0 con 0x7f4a80068960 2023-09-30T00:53:15.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 10 librados: watch_flush enter 2023-09-30T00:53:15.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 10 librados: watch_flush exit 2023-09-30T00:53:15.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 20 client.4966.objecter shutdown clearing up homeless session... 2023-09-30T00:53:15.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 10 client.4966.objecter successfully canceled tick 2023-09-30T00:53:15.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 1 -- 172.21.15.40:0/884766622 >> v2:172.21.15.135:6800/116426 conn(0x7f4a80068960 msgr2=0x7f4a8006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 1 --2- 172.21.15.40:0/884766622 >> v2:172.21.15.135:6800/116426 conn(0x7f4a80068960 0x7f4a8006ae10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4a840091e0 tx=0x7f4a84017000 comp rx=0 tx=0).stop 2023-09-30T00:53:15.551 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 10 monclient: shutdown 2023-09-30T00:53:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 1 -- 172.21.15.40:0/884766622 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a940af410 msgr2=0x7f4a94133dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 1 --2- 172.21.15.40:0/884766622 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a940af410 0x7f4a94133dc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800ec80 tx=0x7f4a880091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 1 -- 172.21.15.40:0/884766622 shutdown_connections 2023-09-30T00:53:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 1 --2- 172.21.15.40:0/884766622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4a940b1460 0x7f4a94134300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 1 --2- 172.21.15.40:0/884766622 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f4a940a5fb0 0x7f4a94151220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 1 --2- 172.21.15.40:0/884766622 >> v2:172.21.15.135:6800/116426 conn(0x7f4a80068960 0x7f4a8006ae10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 1 --2- 172.21.15.40:0/884766622 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f4a940af410 0x7f4a94133dc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 1 -- 172.21.15.40:0/884766622 >> 172.21.15.40:0/884766622 conn(0x7f4a940631f0 msgr2=0x7f4a940b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 1 -- 172.21.15.40:0/884766622 shutdown_connections 2023-09-30T00:53:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 1 -- 172.21.15.40:0/884766622 wait complete. 2023-09-30T00:53:15.552 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.550+0000 7f4a993ff700 1 librados: shutdown 2023-09-30T00:53:15.553 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 5 to deep-scrub 2023-09-30T00:53:15.564 INFO:tasks.ceph:Scrubbing osd.6 2023-09-30T00:53:15.564 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:53:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.702+0000 7f17a2d94700 10 monclient: get_monmap_and_config 2023-09-30T00:53:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.702+0000 7f17a2d94700 10 monclient: build_initial_monmap 2023-09-30T00:53:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.702+0000 7f17a2d94700 10 monclient: monmap: 2023-09-30T00:53:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:15.703727+0000 2023-09-30T00:53:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:15.703727+0000 2023-09-30T00:53:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:15.703 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:15.704 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:15.704 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.703+0000 7f17a2d94700 1 Processor -- start 2023-09-30T00:53:15.705 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.703+0000 7f17a2d94700 1 -- start start 2023-09-30T00:53:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.703+0000 7f17a2d94700 10 monclient: init 2023-09-30T00:53:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a2d94700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a2d94700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-09-30T00:53:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a2d94700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f179c0a5710 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a2d94700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f179c0a7630 0x7f179c0a7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a2d94700 10 monclient(hunting): picked mon.noname-c con 0x7f179c0a7630 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a2d94700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f179c0a7fc0 0x7f179c13af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a2d94700 10 monclient(hunting): picked mon.noname-a con 0x7f179c0a7fc0 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:15.706 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a2d94700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179c0b1770 con 0x7f179c0a5710 2023-09-30T00:53:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a2d94700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a2d94700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f179c0a5ae0 con 0x7f179c0a7fc0 2023-09-30T00:53:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a2d94700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a2d94700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f179c0b1440 con 0x7f179c0a7630 2023-09-30T00:53:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a2d94700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a2d94700 10 monclient(hunting): authenticate will time out at 5657.005306s 2023-09-30T00:53:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.704+0000 7f17a1331700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f179c0a7fc0 0x7f179c13af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.707 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.705+0000 7f179bfff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f179c0a7630 0x7f179c0a7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.705+0000 7f17a1331700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f179c0a7fc0 0x7f179c13af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46192/0 (socket says 172.21.15.40:46192) 2023-09-30T00:53:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.705+0000 7f17a1331700 1 -- 172.21.15.40:0/946089313 learned_addr learned my addr 172.21.15.40:0/946089313 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.705+0000 7f17a1331700 10 monclient(hunting): get_auth_request con 0x7f179c0a7fc0 auth_method 0 2023-09-30T00:53:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.705+0000 7f17a1331700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.705+0000 7f17a1331700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.708 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.705+0000 7f17a1331700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.705+0000 7f179bfff700 10 monclient(hunting): get_auth_request con 0x7f179c0a7630 auth_method 0 2023-09-30T00:53:15.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.705+0000 7f179bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:15.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.705+0000 7f179bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.705+0000 7f179bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.706+0000 7f17a1331700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:15.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.706+0000 7f17a1331700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:15.709 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.706+0000 7f17a1331700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.706+0000 7f179bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.706+0000 7f179bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.706+0000 7f179bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.708+0000 7f17a1331700 10 monclient(hunting): handle_auth_done global_id 4887 payload 274 2023-09-30T00:53:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.708+0000 7f17a1331700 1 -- 172.21.15.40:0/946089313 >> v2:172.21.15.135:3300/0 conn(0x7f179c0a7630 msgr2=0x7f179c0a7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.708+0000 7f17a1331700 1 --2- 172.21.15.40:0/946089313 >> v2:172.21.15.135:3300/0 conn(0x7f179c0a7630 0x7f179c0a7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.708+0000 7f17a1331700 1 -- 172.21.15.40:0/946089313 >> v1:172.21.15.135:6789/0 conn(0x7f179c0a5710 legacy=0x7f179c0b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.708+0000 7f17a1331700 10 monclient: _finish_hunting 0 2023-09-30T00:53:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.708+0000 7f17a1331700 1 monclient: found mon.noname-a 2023-09-30T00:53:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.708+0000 7f17a1331700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:15.710 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.708+0000 7f17a1331700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:53:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.708+0000 7f17a1331700 1 -- 172.21.15.40:0/946089313 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179c0a7210 con 0x7f179c0a7fc0 2023-09-30T00:53:15.711 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.710+0000 7f17a1331700 1 --2- 172.21.15.40:0/946089313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f179c0a7fc0 0x7f179c13af10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f179c064130 tx=0x7f1794009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d16cec2f46d8df server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f179b7fe700 1 -- 172.21.15.40:0/946089313 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179400e070 con 0x7f179c0a7fc0 2023-09-30T00:53:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f179b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:15.715 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f179b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:53:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f179b7fe700 10 monclient: dump: 2023-09-30T00:53:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:15.716 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f179b7fe700 10 monclient: _finish_auth 0 2023-09-30T00:53:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f179b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f179b7fe700 1 -- 172.21.15.40:0/946089313 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179400ba20 con 0x7f179c0a7fc0 2023-09-30T00:53:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f179b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f179b7fe700 1 -- 172.21.15.40:0/946089313 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17940092e0 con 0x7f179c0a7fc0 2023-09-30T00:53:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f17a2d94700 5 monclient: authenticate success, global_id 4887 2023-09-30T00:53:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f17a2d94700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f17a2d94700 4 set_mon_vals no callback set 2023-09-30T00:53:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f179b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f179b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:53:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f179b7fe700 10 monclient: dump: 2023-09-30T00:53:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:15.717 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.718 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.718 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:15.718 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:15.718 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:15.718 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:15.718 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:15.718 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f17a2d94700 10 monclient: shutdown 2023-09-30T00:53:15.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f17a2d94700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:15.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f17a2d94700 1 -- 172.21.15.40:0/946089313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f179c0a7fc0 msgr2=0x7f179c13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f17a2d94700 1 --2- 172.21.15.40:0/946089313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f179c0a7fc0 0x7f179c13af10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f179c064130 tx=0x7f1794009840 comp rx=0 tx=0).stop 2023-09-30T00:53:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f17a2d94700 1 -- 172.21.15.40:0/946089313 shutdown_connections 2023-09-30T00:53:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f17a2d94700 1 --2- 172.21.15.40:0/946089313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f179c0a7fc0 0x7f179c13af10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.715+0000 7f17a2d94700 1 --2- 172.21.15.40:0/946089313 >> v2:172.21.15.135:3300/0 conn(0x7f179c0a7630 0x7f179c0a7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.716+0000 7f17a2d94700 1 -- 172.21.15.40:0/946089313 >> 172.21.15.40:0/946089313 conn(0x7f179c069060 msgr2=0x7f179c0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.716+0000 7f17a2d94700 1 -- 172.21.15.40:0/946089313 shutdown_connections 2023-09-30T00:53:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.716+0000 7f17a2d94700 1 -- 172.21.15.40:0/946089313 wait complete. 2023-09-30T00:53:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.716+0000 7f17a2d94700 10 monclient: build_initial_monmap 2023-09-30T00:53:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.716+0000 7f17a2d94700 10 monclient: monmap: 2023-09-30T00:53:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:15.722 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:15.723 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:15.723 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:15.723 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:15.723 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:15.723 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:15.723 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:15.723 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.717+0000 7f17a2d94700 1 librados: starting msgr at 2023-09-30T00:53:15.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.717+0000 7f17a2d94700 1 librados: starting objecter 2023-09-30T00:53:15.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.717+0000 7f17a2d94700 1 Processor -- start 2023-09-30T00:53:15.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.717+0000 7f17a2d94700 1 -- start start 2023-09-30T00:53:15.723 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.717+0000 7f17a2d94700 1 librados: setting wanted keys 2023-09-30T00:53:15.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.717+0000 7f17a2d94700 1 librados: calling monclient init 2023-09-30T00:53:15.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.717+0000 7f17a2d94700 10 monclient: init 2023-09-30T00:53:15.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:15.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:53:15.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f179c0a5710 0x7f179c13a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 10 monclient(hunting): picked mon.noname-b con 0x7f179c0a5710 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:15.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f179c0a7630 0x7f179c13c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 10 monclient(hunting): picked mon.noname-c con 0x7f179c0a7630 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:15.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f179c0a7fc0 0x7f179c13c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 10 monclient(hunting): picked mon.noname-a con 0x7f179c0a7fc0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:15.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c13f110 con 0x7f179c0a7fc0 2023-09-30T00:53:15.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179c13d350 con 0x7f179c0a5710 2023-09-30T00:53:15.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179c13d4c0 con 0x7f179c0a7630 2023-09-30T00:53:15.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:15.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.718+0000 7f17a2d94700 10 monclient(hunting): authenticate will time out at 5657.019535s 2023-09-30T00:53:15.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f17a1331700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f179c0a7fc0 0x7f179c13c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f179bfff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f179c0a7630 0x7f179c13c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f17a0b30700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f179c0a5710 0x7f179c13a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f17a1331700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f179c0a7fc0 0x7f179c13c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46208/0 (socket says 172.21.15.40:46208) 2023-09-30T00:53:15.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f17a1331700 1 -- 172.21.15.40:0/803577083 learned_addr learned my addr 172.21.15.40:0/803577083 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:15.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f17a1331700 10 monclient(hunting): get_auth_request con 0x7f179c0a7fc0 auth_method 0 2023-09-30T00:53:15.725 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f17a1331700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:15.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f17a1331700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f17a1331700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f17a0b30700 10 monclient(hunting): get_auth_request con 0x7f179c0a5710 auth_method 0 2023-09-30T00:53:15.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f17a0b30700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:15.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f17a0b30700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f17a0b30700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f179bfff700 10 monclient(hunting): get_auth_request con 0x7f179c0a7630 auth_method 0 2023-09-30T00:53:15.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f179bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:15.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f179bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.719+0000 7f179bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.726 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.720+0000 7f17a1331700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.720+0000 7f17a1331700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.720+0000 7f17a1331700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.720+0000 7f17a0b30700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.720+0000 7f17a0b30700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.720+0000 7f17a0b30700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.720+0000 7f179bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.720+0000 7f179bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:15.727 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.720+0000 7f179bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.723+0000 7f17a0b30700 10 monclient(hunting): handle_auth_done global_id 4975 payload 824 2023-09-30T00:53:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.723+0000 7f17a0b30700 1 -- 172.21.15.40:0/803577083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f179c0a7630 msgr2=0x7f179c13c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.723+0000 7f17a0b30700 1 --2- 172.21.15.40:0/803577083 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f179c0a7630 0x7f179c13c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.723+0000 7f17a0b30700 1 -- 172.21.15.40:0/803577083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f179c0a7fc0 msgr2=0x7f179c13c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.723+0000 7f17a0b30700 1 --2- 172.21.15.40:0/803577083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f179c0a7fc0 0x7f179c13c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.728 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.723+0000 7f17a0b30700 10 monclient: _finish_hunting 0 2023-09-30T00:53:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.723+0000 7f17a0b30700 1 monclient: found mon.noname-b 2023-09-30T00:53:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.723+0000 7f17a0b30700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.723+0000 7f17a0b30700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.723+0000 7f17a0b30700 1 -- 172.21.15.40:0/803577083 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1794009ae0 con 0x7f179c0a5710 2023-09-30T00:53:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.725+0000 7f17a0b30700 1 --2- 172.21.15.40:0/803577083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f179c0a5710 0x7f179c13a650 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f178c00c5f0 tx=0x7f178c007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.725+0000 7f1798ff9700 10 client.?.objecter ms_handle_connect 0x7f179c0a5710 2023-09-30T00:53:15.729 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.725+0000 7f1798ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:15.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f1798ff9700 1 -- 172.21.15.40:0/803577083 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c011590 con 0x7f179c0a5710 2023-09-30T00:53:15.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f1798ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:15.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f1798ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:15.730 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f1798ff9700 10 monclient: dump: 2023-09-30T00:53:15.730 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:15.731 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f1798ff9700 10 monclient: _finish_auth 0 2023-09-30T00:53:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f1798ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:15.731 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f1798ff9700 1 -- 172.21.15.40:0/803577083 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178c011710 con 0x7f179c0a5710 2023-09-30T00:53:15.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f17a2d94700 5 monclient: authenticate success, global_id 4975 2023-09-30T00:53:15.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f17a2d94700 10 monclient: _renew_subs 2023-09-30T00:53:15.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f17a2d94700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:15.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f17a2d94700 1 -- 172.21.15.40:0/803577083 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f179c13ce50 con 0x7f179c0a5710 2023-09-30T00:53:15.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f1798ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:15.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f1798ff9700 1 -- 172.21.15.40:0/803577083 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c011ad0 con 0x7f179c0a5710 2023-09-30T00:53:15.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f1798ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:15.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f1798ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:15.732 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f1798ff9700 10 monclient: dump: 2023-09-30T00:53:15.732 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:15.733 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:15.733 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.733 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.733 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:15.733 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:15.733 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:15.733 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:15.733 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:15.733 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f179affd700 4 set_mon_vals no callback set 2023-09-30T00:53:15.733 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f17a2d94700 10 client.4975.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:15.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f17a2d94700 10 monclient: _renew_subs 2023-09-30T00:53:15.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f17a2d94700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:15.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f17a2d94700 1 -- 172.21.15.40:0/803577083 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f179c0a87a0 con 0x7f179c0a5710 2023-09-30T00:53:15.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.729+0000 7f17a2d94700 1 librados: init done 2023-09-30T00:53:15.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.730+0000 7f17a2d94700 10 client.4975.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:15.734 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.730+0000 7f17a2d94700 15 client.4975.objecter _session_command_op_assign -1 1 2023-09-30T00:53:15.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.730+0000 7f17a2d94700 20 client.4975.objecter _get_session osd=-1 returning homeless 2023-09-30T00:53:15.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.730+0000 7f17a2d94700 10 client.4975.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:15.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.730+0000 7f17a2d94700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:15.735 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.730+0000 7f17a2d94700 1 -- 172.21.15.40:0/803577083 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f179c13bec0 con 0x7f179c0a5710 2023-09-30T00:53:15.750 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.750+0000 7f1798ff9700 1 -- 172.21.15.40:0/803577083 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f178c01b3b0 con 0x7f179c0a5710 2023-09-30T00:53:15.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.750+0000 7f1798ff9700 1 --2- 172.21.15.40:0/803577083 >> v2:172.21.15.135:6800/116426 conn(0x7f177c068960 0x7f177c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.751+0000 7f179bfff700 1 --2- 172.21.15.40:0/803577083 >> v2:172.21.15.135:6800/116426 conn(0x7f177c068960 0x7f177c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.751 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.751+0000 7f179bfff700 10 monclient: get_auth_request con 0x7f177c068960 auth_method 0 2023-09-30T00:53:15.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.751+0000 7f179bfff700 1 --2- 172.21.15.40:0/803577083 >> v2:172.21.15.135:6800/116426 conn(0x7f177c068960 0x7f177c06ae10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f179c13ad60 tx=0x7f17840091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:15.752 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.751+0000 7f1798ff9700 10 client.4975.objecter ms_handle_connect 0x7f177c068960 2023-09-30T00:53:15.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 1 -- 172.21.15.40:0/803577083 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f178c055580 con 0x7f179c0a5710 2023-09-30T00:53:15.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 10 client.4975.objecter ms_dispatch 0x7f179c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:15.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 3 client.4975.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:15.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 3 client.4975.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:15.755 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 10 client.4975.objecter checking command 1 2023-09-30T00:53:15.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 1 --2- 172.21.15.40:0/803577083 >> v2:172.21.15.135:6813/117133 conn(0x7f177c06e8d0 0x7f177c070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 20 client.4975.objecter _get_session s=0x7f177c06e220 osd=6 3 2023-09-30T00:53:15.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 20 client.4975.objecter put_session s=0x7f177c06e220 osd=6 3 2023-09-30T00:53:15.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 15 client.4975.objecter _session_command_op_remove -1 1 2023-09-30T00:53:15.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 20 client.4975.objecter _get_session s=0x7f177c06e220 osd=6 3 2023-09-30T00:53:15.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 20 client.4975.objecter get_session s=0x7f177c06e220 osd=6 3 2023-09-30T00:53:15.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 15 client.4975.objecter _session_command_op_assign 6 1 2023-09-30T00:53:15.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 20 client.4975.objecter put_session s=0x7f177c06e220 osd=6 4 2023-09-30T00:53:15.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 10 client.4975.objecter _send_command 1 2023-09-30T00:53:15.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 1 -- 172.21.15.40:0/803577083 --> v2:172.21.15.135:6813/117133 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f177c071350 con 0x7f177c06e8d0 2023-09-30T00:53:15.756 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 20 client.4975.objecter dump_active .. 0 homeless 2023-09-30T00:53:15.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 1 -- 172.21.15.40:0/803577083 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f178c001120 con 0x7f179c0a5710 2023-09-30T00:53:15.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f1798ff9700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:53:15.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.755+0000 7f17a1331700 1 --2- 172.21.15.40:0/803577083 >> v2:172.21.15.135:6813/117133 conn(0x7f177c06e8d0 0x7f177c070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.757 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.756+0000 7f17a1331700 10 monclient: get_auth_request con 0x7f177c06e8d0 auth_method 0 2023-09-30T00:53:15.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.759+0000 7f17a1331700 1 --2- 172.21.15.40:0/803577083 >> v2:172.21.15.135:6813/117133 conn(0x7f177c06e8d0 0x7f177c070ce0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:15.759 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.759+0000 7f1798ff9700 10 client.4975.objecter ms_handle_connect 0x7f177c06e8d0 2023-09-30T00:53:15.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.781+0000 7f1798ff9700 1 -- 172.21.15.40:0/803577083 <== osd.6 v2:172.21.15.135:6813/117133 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f177c071350 con 0x7f177c06e8d0 2023-09-30T00:53:15.781 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.781+0000 7f1798ff9700 10 client.4975.objecter ms_dispatch 0x7f179c001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:53:15.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.781+0000 7f1798ff9700 10 client.4975.objecter _finish_command 1 = system:0 2023-09-30T00:53:15.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.781+0000 7f1798ff9700 20 client.4975.objecter put_session s=0x7f177c06e220 osd=6 4 2023-09-30T00:53:15.782 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.781+0000 7f1798ff9700 15 client.4975.objecter _session_command_op_remove 6 1 2023-09-30T00:53:15.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.796+0000 7f17a2d94700 10 client.4975.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T00:53:15.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.796+0000 7f17a2d94700 15 client.4975.objecter _session_command_op_assign -1 2 2023-09-30T00:53:15.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.796+0000 7f17a2d94700 20 client.4975.objecter _get_session s=0x7f177c06e220 osd=6 3 2023-09-30T00:53:15.796 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.796+0000 7f17a2d94700 20 client.4975.objecter put_session s=0x7f177c06e220 osd=6 3 2023-09-30T00:53:15.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.796+0000 7f17a2d94700 20 client.4975.objecter _get_session s=0x7f177c06e220 osd=6 3 2023-09-30T00:53:15.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.796+0000 7f17a2d94700 15 client.4975.objecter _session_command_op_remove -1 2 2023-09-30T00:53:15.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.796+0000 7f17a2d94700 20 client.4975.objecter get_session s=0x7f177c06e220 osd=6 3 2023-09-30T00:53:15.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.796+0000 7f17a2d94700 15 client.4975.objecter _session_command_op_assign 6 2 2023-09-30T00:53:15.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.796+0000 7f17a2d94700 20 client.4975.objecter put_session s=0x7f177c06e220 osd=6 4 2023-09-30T00:53:15.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.796+0000 7f17a2d94700 10 client.4975.objecter _send_command 2 2023-09-30T00:53:15.797 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.796+0000 7f17a2d94700 1 -- 172.21.15.40:0/803577083 --> v2:172.21.15.135:6813/117133 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f179c0b1000 con 0x7f177c06e8d0 2023-09-30T00:53:15.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.817+0000 7f1798ff9700 1 -- 172.21.15.40:0/803577083 <== osd.6 v2:172.21.15.135:6813/117133 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f179c0b1000 con 0x7f177c06e8d0 2023-09-30T00:53:15.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.817+0000 7f1798ff9700 10 client.4975.objecter ms_dispatch 0x7f179c001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:53:15.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.817+0000 7f1798ff9700 10 client.4975.objecter _finish_command 2 = system:0 2023-09-30T00:53:15.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.817+0000 7f1798ff9700 20 client.4975.objecter put_session s=0x7f177c06e220 osd=6 4 2023-09-30T00:53:15.818 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.817+0000 7f1798ff9700 15 client.4975.objecter _session_command_op_remove 6 2 2023-09-30T00:53:15.819 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-09-30T00:53:15.819 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:53:15.819 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-09-30T00:53:15.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 10 librados: watch_flush enter 2023-09-30T00:53:15.820 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 10 librados: watch_flush exit 2023-09-30T00:53:15.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 10 client.4975.objecter close_session for osd.6 2023-09-30T00:53:15.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 1 -- 172.21.15.40:0/803577083 >> v2:172.21.15.135:6813/117133 conn(0x7f177c06e8d0 msgr2=0x7f177c070ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 1 --2- 172.21.15.40:0/803577083 >> v2:172.21.15.135:6813/117133 conn(0x7f177c06e8d0 0x7f177c070ce0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 20 client.4975.objecter put_session s=0x7f177c06e220 osd=6 1 2023-09-30T00:53:15.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 20 client.4975.objecter shutdown clearing up homeless session... 2023-09-30T00:53:15.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 10 client.4975.objecter successfully canceled tick 2023-09-30T00:53:15.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 1 -- 172.21.15.40:0/803577083 >> v2:172.21.15.135:6800/116426 conn(0x7f177c068960 msgr2=0x7f177c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 1 --2- 172.21.15.40:0/803577083 >> v2:172.21.15.135:6800/116426 conn(0x7f177c068960 0x7f177c06ae10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f179c13ad60 tx=0x7f17840091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 10 monclient: shutdown 2023-09-30T00:53:15.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:15.821 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 1 -- 172.21.15.40:0/803577083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f179c0a5710 msgr2=0x7f179c13a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 1 --2- 172.21.15.40:0/803577083 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f179c0a5710 0x7f179c13a650 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f178c00c5f0 tx=0x7f178c007ec0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a1331700 1 -- 172.21.15.40:0/803577083 reap_dead start 2023-09-30T00:53:15.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 1 -- 172.21.15.40:0/803577083 shutdown_connections 2023-09-30T00:53:15.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 1 -- 172.21.15.40:0/803577083 >> 172.21.15.40:0/803577083 conn(0x7f179c069060 msgr2=0x7f179c141f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:15.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 1 -- 172.21.15.40:0/803577083 shutdown_connections 2023-09-30T00:53:15.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 1 -- 172.21.15.40:0/803577083 wait complete. 2023-09-30T00:53:15.822 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.818+0000 7f17a2d94700 1 librados: shutdown 2023-09-30T00:53:15.832 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-09-30T00:53:15.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.970+0000 7f75b2877700 10 monclient: get_monmap_and_config 2023-09-30T00:53:15.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.970+0000 7f75b2877700 10 monclient: build_initial_monmap 2023-09-30T00:53:15.971 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.970+0000 7f75b2877700 10 monclient: monmap: 2023-09-30T00:53:15.971 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:15.971976+0000 2023-09-30T00:53:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:15.971976+0000 2023-09-30T00:53:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:15.972 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:15.973 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.971+0000 7f75b2877700 1 Processor -- start 2023-09-30T00:53:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.971+0000 7f75b2877700 1 -- start start 2023-09-30T00:53:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.971+0000 7f75b2877700 10 monclient: init 2023-09-30T00:53:15.974 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.972+0000 7f75b2877700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.972+0000 7f75b2877700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-09-30T00:53:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.972+0000 7f75b2877700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f75ac0a7a40 0x7f75ac0a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.972+0000 7f75b2877700 10 monclient(hunting): picked mon.noname-c con 0x7f75ac0a7a40 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.972+0000 7f75b2877700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75ac0a6000 0x7f75ac13e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.972+0000 7f75b2877700 10 monclient(hunting): picked mon.noname-b con 0x7f75ac0a6000 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.972+0000 7f75b2877700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75ac13e8c0 addr v1:172.21.15.40:6789/0 2023-09-30T00:53:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.972+0000 7f75b2877700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75ac0a5b70 con 0x7f75ac13e8c0 2023-09-30T00:53:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.972+0000 7f75b2877700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.972+0000 7f75b2877700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75ac0a7e10 con 0x7f75ac0a7a40 2023-09-30T00:53:15.975 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.972+0000 7f75b2877700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.972+0000 7f75b2877700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0a6ff0 con 0x7f75ac0a6000 2023-09-30T00:53:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.972+0000 7f75b2877700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.972+0000 7f75b2877700 10 monclient(hunting): authenticate will time out at 5657.273853s 2023-09-30T00:53:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.973+0000 7f75abfff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f75ac0a7a40 0x7f75ac0a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.973+0000 7f75abfff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f75ac0a7a40 0x7f75ac0a54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:56604/0 (socket says 172.21.15.40:56604) 2023-09-30T00:53:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.973+0000 7f75abfff700 1 -- 172.21.15.40:0/155371450 learned_addr learned my addr 172.21.15.40:0/155371450 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.973+0000 7f75abfff700 10 monclient(hunting): get_auth_request con 0x7f75ac0a7a40 auth_method 0 2023-09-30T00:53:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.973+0000 7f75abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.973+0000 7f75abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.973+0000 7f75abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.974+0000 7f75abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:15.976 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.974+0000 7f75abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:15.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.974+0000 7f75abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:15.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.975+0000 7f75ab7fe700 1 --2- 172.21.15.40:0/155371450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75ac0a6000 0x7f75ac13e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.977 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.976+0000 7f75abfff700 10 monclient(hunting): handle_auth_done global_id 4978 payload 274 2023-09-30T00:53:15.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.976+0000 7f75abfff700 1 -- 172.21.15.40:0/155371450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75ac0a6000 msgr2=0x7f75ac13e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:15.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.976+0000 7f75abfff700 1 --2- 172.21.15.40:0/155371450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75ac0a6000 0x7f75ac13e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.976+0000 7f75abfff700 1 -- 172.21.15.40:0/155371450 >> v1:172.21.15.40:6789/0 conn(0x7f75ac13e8c0 legacy=0x7f75ac13b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:15.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.976+0000 7f75abfff700 10 monclient: _finish_hunting 0 2023-09-30T00:53:15.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.976+0000 7f75abfff700 1 monclient: found mon.noname-c 2023-09-30T00:53:15.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.976+0000 7f75abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:15.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.976+0000 7f75abfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:53:15.978 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.976+0000 7f75abfff700 1 -- 172.21.15.40:0/155371450 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac0ab890 con 0x7f75ac0a7a40 2023-09-30T00:53:15.979 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.978+0000 7f75abfff700 1 --2- 172.21.15.40:0/155371450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75ac0a7a40 0x7f75ac0a54c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f759c0099e0 tx=0x7f759c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec279f788a1286b4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:15.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75aaffd700 1 -- 172.21.15.40:0/155371450 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c00b7e0 con 0x7f75ac0a7a40 2023-09-30T00:53:15.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:15.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75aaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:53:15.983 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75aaffd700 10 monclient: dump: 2023-09-30T00:53:15.983 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:15.983 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:15.983 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.983 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.983 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:15.984 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:15.984 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:15.984 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:15.984 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:15.984 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75aaffd700 10 monclient: _finish_auth 0 2023-09-30T00:53:15.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75aaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:15.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75aaffd700 1 -- 172.21.15.40:0/155371450 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759c00b940 con 0x7f75ac0a7a40 2023-09-30T00:53:15.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75b2877700 5 monclient: authenticate success, global_id 4978 2023-09-30T00:53:15.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75b2877700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:15.984 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75aaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:15.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75b2877700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:15.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75aaffd700 1 -- 172.21.15.40:0/155371450 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c00bd00 con 0x7f75ac0a7a40 2023-09-30T00:53:15.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75b2877700 4 set_mon_vals no callback set 2023-09-30T00:53:15.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75aaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:15.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75aaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:15.985 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.982+0000 7f75aaffd700 10 monclient: dump: 2023-09-30T00:53:15.985 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:15.985 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:15.985 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.985 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:15.985 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:15.985 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:15.986 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:15.986 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:15.986 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:15.986 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.983+0000 7f75b2877700 10 monclient: shutdown 2023-09-30T00:53:15.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.983+0000 7f75b2877700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:15.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.983+0000 7f75b2877700 1 -- 172.21.15.40:0/155371450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75ac0a7a40 msgr2=0x7f75ac0a54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.983+0000 7f75b2877700 1 --2- 172.21.15.40:0/155371450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75ac0a7a40 0x7f75ac0a54c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f759c0099e0 tx=0x7f759c009250 comp rx=0 tx=0).stop 2023-09-30T00:53:15.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.983+0000 7f75b2877700 1 -- 172.21.15.40:0/155371450 shutdown_connections 2023-09-30T00:53:15.988 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.983+0000 7f75b2877700 1 --2- 172.21.15.40:0/155371450 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75ac0a7a40 0x7f75ac0a54c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.983+0000 7f75b2877700 1 --2- 172.21.15.40:0/155371450 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75ac0a6000 0x7f75ac13e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.983+0000 7f75b2877700 1 -- 172.21.15.40:0/155371450 >> 172.21.15.40:0/155371450 conn(0x7f75ac069150 msgr2=0x7f75ac0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.983+0000 7f75b2877700 1 -- 172.21.15.40:0/155371450 shutdown_connections 2023-09-30T00:53:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.983+0000 7f75b2877700 1 -- 172.21.15.40:0/155371450 wait complete. 2023-09-30T00:53:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.984+0000 7f75b2877700 10 monclient: build_initial_monmap 2023-09-30T00:53:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.984+0000 7f75b2877700 10 monclient: monmap: 2023-09-30T00:53:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:15.989 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:15.990 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:15.990 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:15.990 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:15.990 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:15.990 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:15.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.984+0000 7f75b2877700 1 librados: starting msgr at 2023-09-30T00:53:15.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.984+0000 7f75b2877700 1 librados: starting objecter 2023-09-30T00:53:15.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.985+0000 7f75b2877700 1 Processor -- start 2023-09-30T00:53:15.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.985+0000 7f75b2877700 1 -- start start 2023-09-30T00:53:15.990 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.985+0000 7f75b2877700 1 librados: setting wanted keys 2023-09-30T00:53:15.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.985+0000 7f75b2877700 1 librados: calling monclient init 2023-09-30T00:53:15.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.985+0000 7f75b2877700 10 monclient: init 2023-09-30T00:53:15.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:15.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:53:15.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75ac0a6000 0x7f75ac0ab2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 10 monclient(hunting): picked mon.noname-c con 0x7f75ac0a6000 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:15.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75ac0a7a40 0x7f75ac14e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 10 monclient(hunting): picked mon.noname-b con 0x7f75ac0a7a40 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:15.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75ac13e8c0 0x7f75ac0a8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:15.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 10 monclient(hunting): picked mon.noname-a con 0x7f75ac13e8c0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:15.991 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0aaf90 con 0x7f75ac13e8c0 2023-09-30T00:53:15.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0a8830 con 0x7f75ac0a7a40 2023-09-30T00:53:15.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:15.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac0a8ad0 con 0x7f75ac0a6000 2023-09-30T00:53:15.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:15.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75b2877700 10 monclient(hunting): authenticate will time out at 5657.287207s 2023-09-30T00:53:15.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.986+0000 7f75ab7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75ac0a7a40 0x7f75ac14e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.992 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.987+0000 7f75b0e14700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75ac13e8c0 0x7f75ac0a8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.987+0000 7f75ab7fe700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75ac0a7a40 0x7f75ac14e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:56612/0 (socket says 172.21.15.40:56612) 2023-09-30T00:53:15.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.987+0000 7f75b0e14700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75ac13e8c0 0x7f75ac0a8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46228/0 (socket says 172.21.15.40:46228) 2023-09-30T00:53:15.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.987+0000 7f75ab7fe700 1 -- 172.21.15.40:0/2679892607 learned_addr learned my addr 172.21.15.40:0/2679892607 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:15.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.987+0000 7f75ab7fe700 10 monclient(hunting): get_auth_request con 0x7f75ac0a7a40 auth_method 0 2023-09-30T00:53:15.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.987+0000 7f75ab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:15.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.987+0000 7f75ab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.987+0000 7f75ab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.987+0000 7f75b0e14700 10 monclient(hunting): get_auth_request con 0x7f75ac13e8c0 auth_method 0 2023-09-30T00:53:15.994 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.987+0000 7f75b0e14700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.987+0000 7f75b0e14700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.987+0000 7f75b0e14700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.988+0000 7f75b0e14700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.988+0000 7f75b0e14700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.988+0000 7f75b0e14700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.988+0000 7f75ab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.988+0000 7f75ab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.988+0000 7f75ab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.990+0000 7f75abfff700 1 --2- 172.21.15.40:0/2679892607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75ac0a6000 0x7f75ac0ab2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.990+0000 7f75abfff700 10 monclient(hunting): get_auth_request con 0x7f75ac0a6000 auth_method 0 2023-09-30T00:53:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.990+0000 7f75abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:15.995 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.990+0000 7f75abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.990+0000 7f75abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.991+0000 7f75abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.991+0000 7f75abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.991+0000 7f75abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.991+0000 7f75b0e14700 10 monclient(hunting): handle_auth_done global_id 4893 payload 824 2023-09-30T00:53:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.991+0000 7f75b0e14700 1 -- 172.21.15.40:0/2679892607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75ac0a6000 msgr2=0x7f75ac0ab2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.991+0000 7f75b0e14700 1 --2- 172.21.15.40:0/2679892607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75ac0a6000 0x7f75ac0ab2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.991+0000 7f75b0e14700 1 -- 172.21.15.40:0/2679892607 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75ac0a7a40 msgr2=0x7f75ac14e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.991+0000 7f75b0e14700 1 --2- 172.21.15.40:0/2679892607 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75ac0a7a40 0x7f75ac14e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.991+0000 7f75b0e14700 10 monclient: _finish_hunting 0 2023-09-30T00:53:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.991+0000 7f75b0e14700 1 monclient: found mon.noname-a 2023-09-30T00:53:15.996 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.991+0000 7f75b0e14700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:15.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.991+0000 7f75b0e14700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:53:15.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.991+0000 7f75b0e14700 1 -- 172.21.15.40:0/2679892607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c0094f0 con 0x7f75ac13e8c0 2023-09-30T00:53:15.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.996+0000 7f75b0e14700 1 --2- 172.21.15.40:0/2679892607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75ac13e8c0 0x7f75ac0a8ff0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f75a400bad0 tx=0x7f75a4009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:15.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.996+0000 7f7593fff700 10 client.?.objecter ms_handle_connect 0x7f75ac13e8c0 2023-09-30T00:53:15.997 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:15.996+0000 7f7593fff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:16.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f7593fff700 1 -- 172.21.15.40:0/2679892607 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400c760 con 0x7f75ac13e8c0 2023-09-30T00:53:16.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f7593fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:16.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f7593fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:53:16.003 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f7593fff700 10 monclient: dump: 2023-09-30T00:53:16.003 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:16.003 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:16.003 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.003 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.003 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:16.003 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:16.004 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f7593fff700 10 monclient: _finish_auth 0 2023-09-30T00:53:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f7593fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f7593fff700 1 -- 172.21.15.40:0/2679892607 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a400c8c0 con 0x7f75ac13e8c0 2023-09-30T00:53:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f75b2877700 5 monclient: authenticate success, global_id 4893 2023-09-30T00:53:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f75b2877700 10 monclient: _renew_subs 2023-09-30T00:53:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f75b2877700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:53:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f75b2877700 1 -- 172.21.15.40:0/2679892607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ac14ec30 con 0x7f75ac13e8c0 2023-09-30T00:53:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f7593fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f7593fff700 1 -- 172.21.15.40:0/2679892607 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400cc80 con 0x7f75ac13e8c0 2023-09-30T00:53:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f7593fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f7593fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:53:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f7593fff700 10 monclient: dump: 2023-09-30T00:53:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:16.006 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f75aa7fc700 4 set_mon_vals no callback set 2023-09-30T00:53:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f75b2877700 10 client.4893.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f75b2877700 10 monclient: _renew_subs 2023-09-30T00:53:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f75b2877700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:53:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f75b2877700 1 -- 172.21.15.40:0/2679892607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ac0a9770 con 0x7f75ac13e8c0 2023-09-30T00:53:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.002+0000 7f75b2877700 1 librados: init done 2023-09-30T00:53:16.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.003+0000 7f75b2877700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:16.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.003+0000 7f75b2877700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:16.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.003+0000 7f75b2877700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.40:3300/0 2023-09-30T00:53:16.007 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.003+0000 7f75b2877700 1 -- 172.21.15.40:0/2679892607 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75ac0a8cf0 con 0x7f75ac13e8c0 2023-09-30T00:53:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.242+0000 7f7593fff700 1 -- 172.21.15.40:0/2679892607 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f75a4014620 con 0x7f75ac13e8c0 2023-09-30T00:53:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.243+0000 7f7593fff700 1 --2- 172.21.15.40:0/2679892607 >> v2:172.21.15.135:6800/116426 conn(0x7f7598068960 0x7f759806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:16.243 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.243+0000 7f75abfff700 1 --2- 172.21.15.40:0/2679892607 >> v2:172.21.15.135:6800/116426 conn(0x7f7598068960 0x7f759806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:16.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.243+0000 7f75abfff700 10 monclient: get_auth_request con 0x7f7598068960 auth_method 0 2023-09-30T00:53:16.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.243+0000 7f7593fff700 10 client.4893.objecter ms_handle_connect 0x7f7598068960 2023-09-30T00:53:16.244 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.243+0000 7f75abfff700 1 --2- 172.21.15.40:0/2679892607 >> v2:172.21.15.135:6800/116426 conn(0x7f7598068960 0x7f759806ae10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f759c0099e0 tx=0x7f759c016000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:16.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.299+0000 7f7593fff700 1 -- 172.21.15.40:0/2679892607 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f75a4056510 con 0x7f75ac13e8c0 2023-09-30T00:53:16.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.299+0000 7f7593fff700 10 client.4893.objecter ms_dispatch 0x7f75ac001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:16.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.299+0000 7f7593fff700 3 client.4893.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:16.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.299+0000 7f7593fff700 3 client.4893.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:16.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.299+0000 7f7593fff700 20 client.4893.objecter dump_active .. 0 homeless 2023-09-30T00:53:16.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.299+0000 7f7593fff700 1 -- 172.21.15.40:0/2679892607 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f75a4051f80 con 0x7f75ac13e8c0 2023-09-30T00:53:16.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.299+0000 7f7593fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:16.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.299+0000 7f7593fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:16.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.509+0000 7f75b2877700 1 -- 172.21.15.40:0/2679892607 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f75ac064a00 con 0x7f7598068960 2023-09-30T00:53:16.510 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.510+0000 7f7593fff700 1 -- 172.21.15.40:0/2679892607 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f75ac064a00 con 0x7f7598068960 2023-09-30T00:53:16.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 10 librados: watch_flush enter 2023-09-30T00:53:16.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 10 librados: watch_flush exit 2023-09-30T00:53:16.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 20 client.4893.objecter shutdown clearing up homeless session... 2023-09-30T00:53:16.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 10 client.4893.objecter successfully canceled tick 2023-09-30T00:53:16.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 1 -- 172.21.15.40:0/2679892607 >> v2:172.21.15.135:6800/116426 conn(0x7f7598068960 msgr2=0x7f759806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:16.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 1 --2- 172.21.15.40:0/2679892607 >> v2:172.21.15.135:6800/116426 conn(0x7f7598068960 0x7f759806ae10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f759c0099e0 tx=0x7f759c016000 comp rx=0 tx=0).stop 2023-09-30T00:53:16.513 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 10 monclient: shutdown 2023-09-30T00:53:16.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:16.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 1 -- 172.21.15.40:0/2679892607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75ac13e8c0 msgr2=0x7f75ac0a8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:16.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 1 --2- 172.21.15.40:0/2679892607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75ac13e8c0 0x7f75ac0a8ff0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f75a400bad0 tx=0x7f75a4009760 comp rx=0 tx=0).stop 2023-09-30T00:53:16.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 1 -- 172.21.15.40:0/2679892607 shutdown_connections 2023-09-30T00:53:16.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 1 --2- 172.21.15.40:0/2679892607 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75ac13e8c0 0x7f75ac0a8ff0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 1 --2- 172.21.15.40:0/2679892607 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f75ac0a6000 0x7f75ac0ab2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 1 --2- 172.21.15.40:0/2679892607 >> v2:172.21.15.135:6800/116426 conn(0x7f7598068960 0x7f759806ae10 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 1 --2- 172.21.15.40:0/2679892607 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f75ac0a7a40 0x7f75ac14e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 1 -- 172.21.15.40:0/2679892607 >> 172.21.15.40:0/2679892607 conn(0x7f75ac069150 msgr2=0x7f75ac0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:16.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 1 -- 172.21.15.40:0/2679892607 shutdown_connections 2023-09-30T00:53:16.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 1 -- 172.21.15.40:0/2679892607 wait complete. 2023-09-30T00:53:16.514 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.512+0000 7f75b2877700 1 librados: shutdown 2023-09-30T00:53:16.515 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 6 to deep-scrub 2023-09-30T00:53:16.526 INFO:tasks.ceph:Scrubbing osd.7 2023-09-30T00:53:16.526 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:53:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.666+0000 7fb52ba2c700 10 monclient: get_monmap_and_config 2023-09-30T00:53:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.666+0000 7fb52ba2c700 10 monclient: build_initial_monmap 2023-09-30T00:53:16.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.666+0000 7fb52ba2c700 10 monclient: monmap: 2023-09-30T00:53:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:16.667161+0000 2023-09-30T00:53:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:16.667161+0000 2023-09-30T00:53:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:16.667 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:16.667 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.666+0000 7fb52ba2c700 1 Processor -- start 2023-09-30T00:53:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.667+0000 7fb52ba2c700 1 -- start start 2023-09-30T00:53:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.667+0000 7fb52ba2c700 10 monclient: init 2023-09-30T00:53:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.667+0000 7fb52ba2c700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.667+0000 7fb52ba2c700 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-09-30T00:53:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.667+0000 7fb52ba2c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5240a5710 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.667+0000 7fb52ba2c700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fb5240a7630 0x7fb5240a7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.667+0000 7fb52ba2c700 10 monclient(hunting): picked mon.noname-c con 0x7fb5240a7630 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.667+0000 7fb52ba2c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5240a7fc0 addr v1:172.21.15.40:6789/0 2023-09-30T00:53:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.667+0000 7fb52ba2c700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5240b1770 con 0x7fb5240a7fc0 2023-09-30T00:53:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.667+0000 7fb52ba2c700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5240a5ae0 con 0x7fb5240a5710 2023-09-30T00:53:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.667+0000 7fb52ba2c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.667+0000 7fb52ba2c700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5240b1440 con 0x7fb5240a7630 2023-09-30T00:53:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.667+0000 7fb52ba2c700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.667+0000 7fb52ba2c700 10 monclient(hunting): authenticate will time out at 5657.968700s 2023-09-30T00:53:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.668+0000 7fb529fc9700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb5240a7fc0 0x7fb52413af10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53330/0 (socket says 172.21.15.40:53330) 2023-09-30T00:53:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.668+0000 7fb529fc9700 1 -- 172.21.15.40:0/2911162169 learned_addr learned my addr 172.21.15.40:0/2911162169 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:53:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.668+0000 7fb528fc7700 1 --2- 172.21.15.40:0/2911162169 >> v2:172.21.15.135:3300/0 conn(0x7fb5240a7630 0x7fb5240a7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.669+0000 7fb528fc7700 10 monclient(hunting): get_auth_request con 0x7fb5240a7630 auth_method 0 2023-09-30T00:53:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.669+0000 7fb528fc7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.669+0000 7fb528fc7700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.669+0000 7fb528fc7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.670+0000 7fb528fc7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.670+0000 7fb528fc7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.670+0000 7fb528fc7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.671+0000 7fb517fff700 1 -- 172.21.15.40:0/2911162169 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729140949 0 0) 0x7fb5240b1770 con 0x7fb5240a7fc0 2023-09-30T00:53:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.671+0000 7fb517fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.671+0000 7fb517fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.671+0000 7fb517fff700 10 monclient(hunting): my global_id is 4896 2023-09-30T00:53:16.672 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.671+0000 7fb517fff700 1 -- 172.21.15.40:0/2911162169 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb510003570 con 0x7fb5240a7fc0 2023-09-30T00:53:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.672+0000 7fb528fc7700 10 monclient(hunting): handle_auth_done global_id 4984 payload 274 2023-09-30T00:53:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.672+0000 7fb528fc7700 1 -- 172.21.15.40:0/2911162169 >> v1:172.21.15.135:6789/0 conn(0x7fb5240a5710 legacy=0x7fb5240b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.672+0000 7fb528fc7700 1 -- 172.21.15.40:0/2911162169 >> v1:172.21.15.40:6789/0 conn(0x7fb5240a7fc0 legacy=0x7fb52413af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.672+0000 7fb528fc7700 10 monclient: _finish_hunting 0 2023-09-30T00:53:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.672+0000 7fb528fc7700 1 monclient: found mon.noname-c 2023-09-30T00:53:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.672+0000 7fb528fc7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.672+0000 7fb528fc7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:53:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.672+0000 7fb528fc7700 1 -- 172.21.15.40:0/2911162169 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5240a7210 con 0x7fb5240a7630 2023-09-30T00:53:16.675 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.674+0000 7fb528fc7700 1 --2- 172.21.15.40:0/2911162169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5240a7630 0x7fb5240a7a80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb5180098e0 tx=0x7fb5180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=841443eb0c9f35f4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb517fff700 1 -- 172.21.15.40:0/2911162169 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb51800b640 con 0x7fb5240a7630 2023-09-30T00:53:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb517fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb517fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:53:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb517fff700 10 monclient: dump: 2023-09-30T00:53:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:16.682 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:16.683 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb517fff700 10 monclient: _finish_auth 0 2023-09-30T00:53:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb517fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb517fff700 1 -- 172.21.15.40:0/2911162169 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb51800b7a0 con 0x7fb5240a7630 2023-09-30T00:53:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb52ba2c700 5 monclient: authenticate success, global_id 4984 2023-09-30T00:53:16.683 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb52ba2c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb517fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb52ba2c700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb517fff700 1 -- 172.21.15.40:0/2911162169 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb51800bb40 con 0x7fb5240a7630 2023-09-30T00:53:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb517fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb52ba2c700 4 set_mon_vals no callback set 2023-09-30T00:53:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb517fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.681+0000 7fb517fff700 10 monclient: dump: 2023-09-30T00:53:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:16.685 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:16.685 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:16.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.682+0000 7fb52ba2c700 10 monclient: shutdown 2023-09-30T00:53:16.687 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.682+0000 7fb52ba2c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:16.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.682+0000 7fb52ba2c700 1 -- 172.21.15.40:0/2911162169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5240a7630 msgr2=0x7fb5240a7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:16.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.682+0000 7fb52ba2c700 1 --2- 172.21.15.40:0/2911162169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5240a7630 0x7fb5240a7a80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb5180098e0 tx=0x7fb5180091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.682+0000 7fb52ba2c700 1 -- 172.21.15.40:0/2911162169 shutdown_connections 2023-09-30T00:53:16.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.682+0000 7fb52ba2c700 1 --2- 172.21.15.40:0/2911162169 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5240a7630 0x7fb5240a7a80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.682+0000 7fb52ba2c700 1 -- 172.21.15.40:0/2911162169 >> 172.21.15.40:0/2911162169 conn(0x7fb524069060 msgr2=0x7fb5240631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:16.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.682+0000 7fb52ba2c700 1 -- 172.21.15.40:0/2911162169 shutdown_connections 2023-09-30T00:53:16.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.682+0000 7fb52ba2c700 1 -- 172.21.15.40:0/2911162169 wait complete. 2023-09-30T00:53:16.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.683+0000 7fb52ba2c700 10 monclient: build_initial_monmap 2023-09-30T00:53:16.688 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.683+0000 7fb52ba2c700 10 monclient: monmap: 2023-09-30T00:53:16.688 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:16.689 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:16.689 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:16.689 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:16.689 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:16.689 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:16.689 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:16.689 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:16.689 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:16.689 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:16.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.683+0000 7fb52ba2c700 1 librados: starting msgr at 2023-09-30T00:53:16.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.683+0000 7fb52ba2c700 1 librados: starting objecter 2023-09-30T00:53:16.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.683+0000 7fb52ba2c700 1 Processor -- start 2023-09-30T00:53:16.689 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.683+0000 7fb52ba2c700 1 -- start start 2023-09-30T00:53:16.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.683+0000 7fb52ba2c700 1 librados: setting wanted keys 2023-09-30T00:53:16.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.683+0000 7fb52ba2c700 1 librados: calling monclient init 2023-09-30T00:53:16.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.683+0000 7fb52ba2c700 10 monclient: init 2023-09-30T00:53:16.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:16.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:53:16.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5240a5710 0x7fb52413ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:16.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 10 monclient(hunting): picked mon.noname-b con 0x7fb5240a5710 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:16.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5240a7630 0x7fb5240abb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:16.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 10 monclient(hunting): picked mon.noname-c con 0x7fb5240a7630 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:16.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5240a7fc0 0x7fb524140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:16.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 10 monclient(hunting): picked mon.noname-a con 0x7fb5240a7fc0 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:16.690 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5240acfb0 con 0x7fb5240a7fc0 2023-09-30T00:53:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52413e010 con 0x7fb5240a5710 2023-09-30T00:53:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52413d3d0 con 0x7fb5240a7630 2023-09-30T00:53:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.684+0000 7fb52ba2c700 10 monclient(hunting): authenticate will time out at 5657.985770s 2023-09-30T00:53:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.685+0000 7fb529fc9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5240a7fc0 0x7fb524140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.685+0000 7fb528fc7700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5240a7630 0x7fb5240abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.685+0000 7fb5297c8700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5240a5710 0x7fb52413ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.685+0000 7fb529fc9700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5240a7fc0 0x7fb524140dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46240/0 (socket says 172.21.15.40:46240) 2023-09-30T00:53:16.691 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.685+0000 7fb529fc9700 1 -- 172.21.15.40:0/1238043406 learned_addr learned my addr 172.21.15.40:0/1238043406 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.685+0000 7fb529fc9700 10 monclient(hunting): get_auth_request con 0x7fb5240a7fc0 auth_method 0 2023-09-30T00:53:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.685+0000 7fb529fc9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.685+0000 7fb529fc9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.685+0000 7fb529fc9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.685+0000 7fb5297c8700 10 monclient(hunting): get_auth_request con 0x7fb5240a5710 auth_method 0 2023-09-30T00:53:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.685+0000 7fb5297c8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.685+0000 7fb5297c8700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.685+0000 7fb5297c8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.686+0000 7fb528fc7700 10 monclient(hunting): get_auth_request con 0x7fb5240a7630 auth_method 0 2023-09-30T00:53:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.686+0000 7fb528fc7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.686+0000 7fb528fc7700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.686+0000 7fb528fc7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:16.692 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.686+0000 7fb5297c8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.686+0000 7fb5297c8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.686+0000 7fb5297c8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.686+0000 7fb529fc9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.686+0000 7fb529fc9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.686+0000 7fb529fc9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.686+0000 7fb528fc7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.686+0000 7fb528fc7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:16.693 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.686+0000 7fb528fc7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.689+0000 7fb5297c8700 10 monclient(hunting): handle_auth_done global_id 4990 payload 824 2023-09-30T00:53:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.689+0000 7fb5297c8700 1 -- 172.21.15.40:0/1238043406 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5240a7630 msgr2=0x7fb5240abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.689+0000 7fb5297c8700 1 --2- 172.21.15.40:0/1238043406 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fb5240a7630 0x7fb5240abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.689+0000 7fb5297c8700 1 -- 172.21.15.40:0/1238043406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5240a7fc0 msgr2=0x7fb524140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.689+0000 7fb5297c8700 1 --2- 172.21.15.40:0/1238043406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5240a7fc0 0x7fb524140dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.689+0000 7fb5297c8700 10 monclient: _finish_hunting 0 2023-09-30T00:53:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.689+0000 7fb5297c8700 1 monclient: found mon.noname-b 2023-09-30T00:53:16.694 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.689+0000 7fb5297c8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.689+0000 7fb5297c8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.689+0000 7fb5297c8700 1 -- 172.21.15.40:0/1238043406 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb518009420 con 0x7fb5240a5710 2023-09-30T00:53:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.691+0000 7fb5297c8700 1 --2- 172.21.15.40:0/1238043406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5240a5710 0x7fb52413ed50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb52000c590 tx=0x7fb520007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.691+0000 7fb5157fa700 10 client.?.objecter ms_handle_connect 0x7fb5240a5710 2023-09-30T00:53:16.695 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.691+0000 7fb5157fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb5157fa700 1 -- 172.21.15.40:0/1238043406 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520011590 con 0x7fb5240a5710 2023-09-30T00:53:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb5157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb5157fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb5157fa700 10 monclient: dump: 2023-09-30T00:53:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.696 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.697 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:16.697 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:16.697 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:16.697 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:16.697 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:16.697 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:16.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb5157fa700 10 monclient: _finish_auth 0 2023-09-30T00:53:16.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb5157fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:16.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb5157fa700 1 -- 172.21.15.40:0/1238043406 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb520011710 con 0x7fb5240a5710 2023-09-30T00:53:16.697 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb52ba2c700 5 monclient: authenticate success, global_id 4990 2023-09-30T00:53:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb52ba2c700 10 monclient: _renew_subs 2023-09-30T00:53:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb52ba2c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb52ba2c700 1 -- 172.21.15.40:0/1238043406 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5240ad340 con 0x7fb5240a5710 2023-09-30T00:53:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb5157fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb5157fa700 1 -- 172.21.15.40:0/1238043406 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520011ad0 con 0x7fb5240a5710 2023-09-30T00:53:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb5157fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb5157fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb5157fa700 10 monclient: dump: 2023-09-30T00:53:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:16.698 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:16.699 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb5177fe700 4 set_mon_vals no callback set 2023-09-30T00:53:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb52ba2c700 10 client.4990.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:16.699 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb52ba2c700 10 monclient: _renew_subs 2023-09-30T00:53:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb52ba2c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb52ba2c700 1 -- 172.21.15.40:0/1238043406 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5240ad7b0 con 0x7fb5240a5710 2023-09-30T00:53:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.695+0000 7fb52ba2c700 1 librados: init done 2023-09-30T00:53:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.696+0000 7fb52ba2c700 10 client.4990.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.696+0000 7fb52ba2c700 15 client.4990.objecter _session_command_op_assign -1 1 2023-09-30T00:53:16.700 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.696+0000 7fb52ba2c700 20 client.4990.objecter _get_session osd=-1 returning homeless 2023-09-30T00:53:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.696+0000 7fb52ba2c700 10 client.4990.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.696+0000 7fb52ba2c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:16.701 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.696+0000 7fb52ba2c700 1 -- 172.21.15.40:0/1238043406 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb524064a00 con 0x7fb5240a5710 2023-09-30T00:53:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.715+0000 7fb5157fa700 1 -- 172.21.15.40:0/1238043406 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fb52001b3b0 con 0x7fb5240a5710 2023-09-30T00:53:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.715+0000 7fb5157fa700 1 --2- 172.21.15.40:0/1238043406 >> v2:172.21.15.135:6800/116426 conn(0x7fb508068960 0x7fb50806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.715+0000 7fb528fc7700 1 --2- 172.21.15.40:0/1238043406 >> v2:172.21.15.135:6800/116426 conn(0x7fb508068960 0x7fb50806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.715+0000 7fb528fc7700 10 monclient: get_auth_request con 0x7fb508068960 auth_method 0 2023-09-30T00:53:16.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.716+0000 7fb528fc7700 1 --2- 172.21.15.40:0/1238043406 >> v2:172.21.15.135:6800/116426 conn(0x7fb508068960 0x7fb50806ae10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb5180093f0 tx=0x7fb518017000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:16.717 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.716+0000 7fb5157fa700 10 client.4990.objecter ms_handle_connect 0x7fb508068960 2023-09-30T00:53:16.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 1 -- 172.21.15.40:0/1238043406 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fb5200555b0 con 0x7fb5240a5710 2023-09-30T00:53:16.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 10 client.4990.objecter ms_dispatch 0x7fb524001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:16.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 3 client.4990.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:16.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 3 client.4990.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:16.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 10 client.4990.objecter checking command 1 2023-09-30T00:53:16.720 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 1 --2- 172.21.15.40:0/1238043406 >> v2:172.21.15.135:6809/117132 conn(0x7fb50806e8d0 0x7fb508070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:16.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 20 client.4990.objecter _get_session s=0x7fb50806e220 osd=7 3 2023-09-30T00:53:16.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 20 client.4990.objecter put_session s=0x7fb50806e220 osd=7 3 2023-09-30T00:53:16.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 15 client.4990.objecter _session_command_op_remove -1 1 2023-09-30T00:53:16.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 20 client.4990.objecter _get_session s=0x7fb50806e220 osd=7 3 2023-09-30T00:53:16.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 20 client.4990.objecter get_session s=0x7fb50806e220 osd=7 3 2023-09-30T00:53:16.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 15 client.4990.objecter _session_command_op_assign 7 1 2023-09-30T00:53:16.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 20 client.4990.objecter put_session s=0x7fb50806e220 osd=7 4 2023-09-30T00:53:16.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 10 client.4990.objecter _send_command 1 2023-09-30T00:53:16.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 1 -- 172.21.15.40:0/1238043406 --> v2:172.21.15.135:6809/117132 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb508071350 con 0x7fb50806e8d0 2023-09-30T00:53:16.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.719+0000 7fb5157fa700 20 client.4990.objecter dump_active .. 0 homeless 2023-09-30T00:53:16.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.720+0000 7fb5157fa700 1 -- 172.21.15.40:0/1238043406 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb520001120 con 0x7fb5240a5710 2023-09-30T00:53:16.721 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.720+0000 7fb5157fa700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T00:53:16.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.720+0000 7fb529fc9700 1 --2- 172.21.15.40:0/1238043406 >> v2:172.21.15.135:6809/117132 conn(0x7fb50806e8d0 0x7fb508070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:16.722 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.721+0000 7fb529fc9700 10 monclient: get_auth_request con 0x7fb50806e8d0 auth_method 0 2023-09-30T00:53:16.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.723+0000 7fb529fc9700 1 --2- 172.21.15.40:0/1238043406 >> v2:172.21.15.135:6809/117132 conn(0x7fb50806e8d0 0x7fb508070ce0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:16.724 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.723+0000 7fb5157fa700 10 client.4990.objecter ms_handle_connect 0x7fb50806e8d0 2023-09-30T00:53:16.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.746+0000 7fb5157fa700 1 -- 172.21.15.40:0/1238043406 <== osd.7 v2:172.21.15.135:6809/117132 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fb508071350 con 0x7fb50806e8d0 2023-09-30T00:53:16.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.746+0000 7fb5157fa700 10 client.4990.objecter ms_dispatch 0x7fb524001550 command_reply(tid 1: 0 ) v1 2023-09-30T00:53:16.746 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.746+0000 7fb5157fa700 10 client.4990.objecter _finish_command 1 = system:0 2023-09-30T00:53:16.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.746+0000 7fb5157fa700 20 client.4990.objecter put_session s=0x7fb50806e220 osd=7 4 2023-09-30T00:53:16.747 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.746+0000 7fb5157fa700 15 client.4990.objecter _session_command_op_remove 7 1 2023-09-30T00:53:16.763 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.763+0000 7fb52ba2c700 10 client.4990.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T00:53:16.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.763+0000 7fb52ba2c700 15 client.4990.objecter _session_command_op_assign -1 2 2023-09-30T00:53:16.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.763+0000 7fb52ba2c700 20 client.4990.objecter _get_session s=0x7fb50806e220 osd=7 3 2023-09-30T00:53:16.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.763+0000 7fb52ba2c700 20 client.4990.objecter put_session s=0x7fb50806e220 osd=7 3 2023-09-30T00:53:16.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.763+0000 7fb52ba2c700 20 client.4990.objecter _get_session s=0x7fb50806e220 osd=7 3 2023-09-30T00:53:16.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.763+0000 7fb52ba2c700 15 client.4990.objecter _session_command_op_remove -1 2 2023-09-30T00:53:16.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.763+0000 7fb52ba2c700 20 client.4990.objecter get_session s=0x7fb50806e220 osd=7 3 2023-09-30T00:53:16.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.763+0000 7fb52ba2c700 15 client.4990.objecter _session_command_op_assign 7 2 2023-09-30T00:53:16.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.763+0000 7fb52ba2c700 20 client.4990.objecter put_session s=0x7fb50806e220 osd=7 4 2023-09-30T00:53:16.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.763+0000 7fb52ba2c700 10 client.4990.objecter _send_command 2 2023-09-30T00:53:16.764 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.763+0000 7fb52ba2c700 1 -- 172.21.15.40:0/1238043406 --> v2:172.21.15.135:6809/117132 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb5240adc10 con 0x7fb50806e8d0 2023-09-30T00:53:16.785 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.785+0000 7fb5157fa700 1 -- 172.21.15.40:0/1238043406 <== osd.7 v2:172.21.15.135:6809/117132 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fb5240adc10 con 0x7fb50806e8d0 2023-09-30T00:53:16.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.785+0000 7fb5157fa700 10 client.4990.objecter ms_dispatch 0x7fb524001550 command_reply(tid 2: 0 ) v1 2023-09-30T00:53:16.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.785+0000 7fb5157fa700 10 client.4990.objecter _finish_command 2 = system:0 2023-09-30T00:53:16.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.785+0000 7fb5157fa700 20 client.4990.objecter put_session s=0x7fb50806e220 osd=7 4 2023-09-30T00:53:16.786 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.785+0000 7fb5157fa700 15 client.4990.objecter _session_command_op_remove 7 2 2023-09-30T00:53:16.786 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2023-09-30T00:53:16.786 INFO:teuthology.orchestra.run.smithi040.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:53:16.786 INFO:teuthology.orchestra.run.smithi040.stdout:} 2023-09-30T00:53:16.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 10 librados: watch_flush enter 2023-09-30T00:53:16.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 10 librados: watch_flush exit 2023-09-30T00:53:16.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 10 client.4990.objecter close_session for osd.7 2023-09-30T00:53:16.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 1 -- 172.21.15.40:0/1238043406 >> v2:172.21.15.135:6809/117132 conn(0x7fb50806e8d0 msgr2=0x7fb508070ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:16.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 1 --2- 172.21.15.40:0/1238043406 >> v2:172.21.15.135:6809/117132 conn(0x7fb50806e8d0 0x7fb508070ce0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 20 client.4990.objecter put_session s=0x7fb50806e220 osd=7 1 2023-09-30T00:53:16.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 20 client.4990.objecter shutdown clearing up homeless session... 2023-09-30T00:53:16.788 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 10 client.4990.objecter successfully canceled tick 2023-09-30T00:53:16.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 1 -- 172.21.15.40:0/1238043406 >> v2:172.21.15.135:6800/116426 conn(0x7fb508068960 msgr2=0x7fb50806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:16.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 1 --2- 172.21.15.40:0/1238043406 >> v2:172.21.15.135:6800/116426 conn(0x7fb508068960 0x7fb50806ae10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb5180093f0 tx=0x7fb518017000 comp rx=0 tx=0).stop 2023-09-30T00:53:16.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 10 monclient: shutdown 2023-09-30T00:53:16.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:16.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 1 -- 172.21.15.40:0/1238043406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5240a5710 msgr2=0x7fb52413ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:16.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 1 --2- 172.21.15.40:0/1238043406 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fb5240a5710 0x7fb52413ed50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb52000c590 tx=0x7fb520007ec0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb529fc9700 1 -- 172.21.15.40:0/1238043406 reap_dead start 2023-09-30T00:53:16.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 1 -- 172.21.15.40:0/1238043406 shutdown_connections 2023-09-30T00:53:16.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 1 -- 172.21.15.40:0/1238043406 >> 172.21.15.40:0/1238043406 conn(0x7fb524069060 msgr2=0x7fb5240b49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:16.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 1 -- 172.21.15.40:0/1238043406 shutdown_connections 2023-09-30T00:53:16.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 1 -- 172.21.15.40:0/1238043406 wait complete. 2023-09-30T00:53:16.789 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.786+0000 7fb52ba2c700 1 librados: shutdown 2023-09-30T00:53:16.800 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-09-30T00:53:16.934 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.934+0000 7f9db3a4a700 10 monclient: get_monmap_and_config 2023-09-30T00:53:16.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.934+0000 7f9db3a4a700 10 monclient: build_initial_monmap 2023-09-30T00:53:16.935 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.934+0000 7f9db3a4a700 10 monclient: monmap: 2023-09-30T00:53:16.935 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:16.935 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:16.935 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:16.935537+0000 2023-09-30T00:53:16.935 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:16.935537+0000 2023-09-30T00:53:16.935 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:16.936 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:16.936 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:16.936 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:16.936 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:16.936 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:16.936 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:16.936 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.935+0000 7f9db3a4a700 1 Processor -- start 2023-09-30T00:53:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.935+0000 7f9db3a4a700 1 -- start start 2023-09-30T00:53:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.935+0000 7f9db3a4a700 10 monclient: init 2023-09-30T00:53:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.935+0000 7f9db3a4a700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.935+0000 7f9db3a4a700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-09-30T00:53:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.935+0000 7f9db3a4a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dac0b1460 0x7f9dac0b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.935+0000 7f9db3a4a700 10 monclient(hunting): picked mon.noname-b con 0x7f9dac0b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:16.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.935+0000 7f9db3a4a700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f9dac0a5fb0 0x7f9dac141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:16.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.936+0000 7f9db3a4a700 10 monclient(hunting): picked mon.noname-c con 0x7f9dac0a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:16.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.936+0000 7f9db3a4a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9dac0af410 addr v1:172.21.15.40:6789/0 2023-09-30T00:53:16.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.936+0000 7f9db3a4a700 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dac0a71f0 con 0x7f9dac0af410 2023-09-30T00:53:16.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.936+0000 7f9db3a4a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:16.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.936+0000 7f9db3a4a700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dac0a74d0 con 0x7f9dac0a5fb0 2023-09-30T00:53:16.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.936+0000 7f9db3a4a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:16.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.936+0000 7f9db3a4a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac0a63c0 con 0x7f9dac0b1460 2023-09-30T00:53:16.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.936+0000 7f9db3a4a700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:16.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.936+0000 7f9db3a4a700 10 monclient(hunting): authenticate will time out at 5658.237049s 2023-09-30T00:53:16.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.936+0000 7f9db1fe7700 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9dac0af410 0x7f9dac0af890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53338/0 (socket says 172.21.15.40:53338) 2023-09-30T00:53:16.939 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.936+0000 7f9db1fe7700 1 -- 172.21.15.40:0/2750887581 learned_addr learned my addr 172.21.15.40:0/2750887581 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:53:16.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.936+0000 7f9db0fe5700 1 --2- 172.21.15.40:0/2750887581 >> v2:172.21.15.135:3300/0 conn(0x7f9dac0a5fb0 0x7f9dac141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:16.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.936+0000 7f9db17e6700 1 --2- 172.21.15.40:0/2750887581 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dac0b1460 0x7f9dac0b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:16.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.937+0000 7f9db0fe5700 10 monclient(hunting): get_auth_request con 0x7f9dac0a5fb0 auth_method 0 2023-09-30T00:53:16.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.937+0000 7f9db0fe5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:16.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.937+0000 7f9db0fe5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:16.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.937+0000 7f9db0fe5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:16.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.937+0000 7f9db17e6700 10 monclient(hunting): get_auth_request con 0x7f9dac0b1460 auth_method 0 2023-09-30T00:53:16.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.937+0000 7f9db17e6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:16.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.937+0000 7f9db17e6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:16.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.937+0000 7f9db17e6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:16.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.937+0000 7f9db0fe5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:16.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.937+0000 7f9db0fe5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.937+0000 7f9db0fe5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.937+0000 7f9db17e6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.937+0000 7f9db17e6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.937+0000 7f9db17e6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.939+0000 7f9db0fe5700 10 monclient(hunting): handle_auth_done global_id 4993 payload 274 2023-09-30T00:53:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.939+0000 7f9db0fe5700 1 -- 172.21.15.40:0/2750887581 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dac0b1460 msgr2=0x7f9dac0b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.939+0000 7f9db0fe5700 1 --2- 172.21.15.40:0/2750887581 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dac0b1460 0x7f9dac0b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.939+0000 7f9db0fe5700 1 -- 172.21.15.40:0/2750887581 >> v1:172.21.15.40:6789/0 conn(0x7f9dac0af410 legacy=0x7f9dac0af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.939+0000 7f9db0fe5700 10 monclient: _finish_hunting 0 2023-09-30T00:53:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.939+0000 7f9db0fe5700 1 monclient: found mon.noname-c 2023-09-30T00:53:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.939+0000 7f9db0fe5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.939+0000 7f9db0fe5700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:53:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.939+0000 7f9db0fe5700 1 -- 172.21.15.40:0/2750887581 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac0a56d0 con 0x7f9dac0a5fb0 2023-09-30T00:53:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.941+0000 7f9db0fe5700 1 --2- 172.21.15.40:0/2750887581 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dac0a5fb0 0x7f9dac141730 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c0098e0 tx=0x7f9d9c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f020a0001b64c0d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:16.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9da3fff700 1 -- 172.21.15.40:0/2750887581 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c00b640 con 0x7f9dac0a5fb0 2023-09-30T00:53:16.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9da3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:16.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9da3fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:53:16.946 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9da3fff700 10 monclient: dump: 2023-09-30T00:53:16.946 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:16.946 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:16.946 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.946 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.947 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:16.947 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:16.947 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:16.947 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:16.947 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:16.947 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:16.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9da3fff700 10 monclient: _finish_auth 0 2023-09-30T00:53:16.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9da3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:16.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9da3fff700 1 -- 172.21.15.40:0/2750887581 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9c00b7a0 con 0x7f9dac0a5fb0 2023-09-30T00:53:16.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9db3a4a700 5 monclient: authenticate success, global_id 4993 2023-09-30T00:53:16.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9db3a4a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:16.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9da3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:16.947 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9db3a4a700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9da3fff700 1 -- 172.21.15.40:0/2750887581 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c00bb40 con 0x7f9dac0a5fb0 2023-09-30T00:53:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9db3a4a700 4 set_mon_vals no callback set 2023-09-30T00:53:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9da3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9da3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.945+0000 7f9da3fff700 10 monclient: dump: 2023-09-30T00:53:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:16.949 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:16.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.946+0000 7f9db3a4a700 10 monclient: shutdown 2023-09-30T00:53:16.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.946+0000 7f9db3a4a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:16.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.946+0000 7f9db3a4a700 1 -- 172.21.15.40:0/2750887581 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dac0a5fb0 msgr2=0x7f9dac141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:16.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.946+0000 7f9db3a4a700 1 --2- 172.21.15.40:0/2750887581 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dac0a5fb0 0x7f9dac141730 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c0098e0 tx=0x7f9d9c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.946+0000 7f9db3a4a700 1 -- 172.21.15.40:0/2750887581 shutdown_connections 2023-09-30T00:53:16.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.946+0000 7f9db3a4a700 1 --2- 172.21.15.40:0/2750887581 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dac0b1460 0x7f9dac0b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.946+0000 7f9db3a4a700 1 --2- 172.21.15.40:0/2750887581 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dac0a5fb0 0x7f9dac141730 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.946+0000 7f9db3a4a700 1 -- 172.21.15.40:0/2750887581 >> 172.21.15.40:0/2750887581 conn(0x7f9dac0631f0 msgr2=0x7f9dac0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:16.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.946+0000 7f9db3a4a700 1 -- 172.21.15.40:0/2750887581 shutdown_connections 2023-09-30T00:53:16.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.946+0000 7f9db3a4a700 1 -- 172.21.15.40:0/2750887581 wait complete. 2023-09-30T00:53:16.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.947+0000 7f9db3a4a700 10 monclient: build_initial_monmap 2023-09-30T00:53:16.953 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.947+0000 7f9db3a4a700 10 monclient: monmap: 2023-09-30T00:53:16.954 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:16.954 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:16.954 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:16.954 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:16.954 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:16.954 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:16.954 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:16.954 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:16.954 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:16.954 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:16.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.947+0000 7f9db3a4a700 1 librados: starting msgr at 2023-09-30T00:53:16.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.947+0000 7f9db3a4a700 1 librados: starting objecter 2023-09-30T00:53:16.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.947+0000 7f9db3a4a700 1 Processor -- start 2023-09-30T00:53:16.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.947+0000 7f9db3a4a700 1 -- start start 2023-09-30T00:53:16.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.947+0000 7f9db3a4a700 1 librados: setting wanted keys 2023-09-30T00:53:16.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.947+0000 7f9db3a4a700 1 librados: calling monclient init 2023-09-30T00:53:16.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.947+0000 7f9db3a4a700 10 monclient: init 2023-09-30T00:53:16.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:16.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:53:16.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dac0a5fb0 0x7f9dac0a8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:16.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 10 monclient(hunting): picked mon.noname-c con 0x7f9dac0a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:16.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dac0af410 0x7f9dac0a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:16.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 10 monclient(hunting): picked mon.noname-a con 0x7f9dac0af410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:16.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dac0b1460 0x7f9dac13cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:16.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 10 monclient(hunting): picked mon.noname-b con 0x7f9dac0b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:16.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:16.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac0a9470 con 0x7f9dac0af410 2023-09-30T00:53:16.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:16.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac0aa5d0 con 0x7f9dac0b1460 2023-09-30T00:53:16.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:16.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac0a8f50 con 0x7f9dac0a5fb0 2023-09-30T00:53:16.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:16.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.948+0000 7f9db3a4a700 10 monclient(hunting): authenticate will time out at 5658.249695s 2023-09-30T00:53:16.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db0fe5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dac0af410 0x7f9dac0a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:16.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db17e6700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dac0a5fb0 0x7f9dac0a8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:16.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db1fe7700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dac0b1460 0x7f9dac13cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:16.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db0fe5700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dac0af410 0x7f9dac0a81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46242/0 (socket says 172.21.15.40:46242) 2023-09-30T00:53:16.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db0fe5700 1 -- 172.21.15.40:0/875673710 learned_addr learned my addr 172.21.15.40:0/875673710 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:16.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db0fe5700 10 monclient(hunting): get_auth_request con 0x7f9dac0af410 auth_method 0 2023-09-30T00:53:16.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db0fe5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:16.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db0fe5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:16.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db0fe5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:16.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db1fe7700 10 monclient(hunting): get_auth_request con 0x7f9dac0b1460 auth_method 0 2023-09-30T00:53:16.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db1fe7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:16.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db1fe7700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:16.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db1fe7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:16.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db17e6700 10 monclient(hunting): get_auth_request con 0x7f9dac0a5fb0 auth_method 0 2023-09-30T00:53:16.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db17e6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:16.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db17e6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:16.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.949+0000 7f9db17e6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:16.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.950+0000 7f9db1fe7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:16.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.950+0000 7f9db1fe7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:16.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.950+0000 7f9db1fe7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:16.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.950+0000 7f9db0fe5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:16.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.950+0000 7f9db0fe5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:16.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.950+0000 7f9db0fe5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:16.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.950+0000 7f9db17e6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:16.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.950+0000 7f9db17e6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:16.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.950+0000 7f9db17e6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:16.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.953+0000 7f9db1fe7700 10 monclient(hunting): handle_auth_done global_id 4996 payload 824 2023-09-30T00:53:16.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.953+0000 7f9db1fe7700 1 -- 172.21.15.40:0/875673710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dac0a5fb0 msgr2=0x7f9dac0a8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:16.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.953+0000 7f9db1fe7700 1 --2- 172.21.15.40:0/875673710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dac0a5fb0 0x7f9dac0a8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.953+0000 7f9db1fe7700 1 -- 172.21.15.40:0/875673710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dac0af410 msgr2=0x7f9dac0a81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:16.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.953+0000 7f9db1fe7700 1 --2- 172.21.15.40:0/875673710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dac0af410 0x7f9dac0a81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:16.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.953+0000 7f9db1fe7700 10 monclient: _finish_hunting 0 2023-09-30T00:53:16.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.953+0000 7f9db1fe7700 1 monclient: found mon.noname-b 2023-09-30T00:53:16.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.953+0000 7f9db1fe7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:16.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.953+0000 7f9db1fe7700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:16.962 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.953+0000 7f9db1fe7700 1 -- 172.21.15.40:0/875673710 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c009420 con 0x7f9dac0b1460 2023-09-30T00:53:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.955+0000 7f9db1fe7700 1 --2- 172.21.15.40:0/875673710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dac0b1460 0x7f9dac13cd30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9da400cb90 tx=0x7f9da40019a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.955+0000 7f9da17fa700 10 client.?.objecter ms_handle_connect 0x7f9dac0b1460 2023-09-30T00:53:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.955+0000 7f9da17fa700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9da17fa700 1 -- 172.21.15.40:0/875673710 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da400a7e0 con 0x7f9dac0b1460 2023-09-30T00:53:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9da17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9da17fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9da17fa700 10 monclient: dump: 2023-09-30T00:53:16.963 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:16.964 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:16.964 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9da17fa700 10 monclient: _finish_auth 0 2023-09-30T00:53:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9da17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9da17fa700 1 -- 172.21.15.40:0/875673710 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da400a980 con 0x7f9dac0b1460 2023-09-30T00:53:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9db3a4a700 5 monclient: authenticate success, global_id 4996 2023-09-30T00:53:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9db3a4a700 10 monclient: _renew_subs 2023-09-30T00:53:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9db3a4a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9db3a4a700 1 -- 172.21.15.40:0/875673710 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dac13ec10 con 0x7f9dac0b1460 2023-09-30T00:53:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9da17fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9da17fa700 1 -- 172.21.15.40:0/875673710 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da400ad40 con 0x7f9dac0b1460 2023-09-30T00:53:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9da17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:16.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9da17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9da17fa700 10 monclient: dump: 2023-09-30T00:53:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:16.966 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:16.967 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9da37fe700 4 set_mon_vals no callback set 2023-09-30T00:53:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9db3a4a700 10 client.4996.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9db3a4a700 10 monclient: _renew_subs 2023-09-30T00:53:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9db3a4a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9db3a4a700 1 -- 172.21.15.40:0/875673710 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dac13d430 con 0x7f9dac0b1460 2023-09-30T00:53:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.959+0000 7f9db3a4a700 1 librados: init done 2023-09-30T00:53:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.960+0000 7f9db3a4a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.960+0000 7f9db3a4a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:16.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.960+0000 7f9db3a4a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:16.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:16.960+0000 7f9db3a4a700 1 -- 172.21.15.40:0/875673710 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dac0a9190 con 0x7f9dac0b1460 2023-09-30T00:53:17.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.211+0000 7f9da17fa700 1 -- 172.21.15.40:0/875673710 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9da4007300 con 0x7f9dac0b1460 2023-09-30T00:53:17.212 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.212+0000 7f9da17fa700 1 --2- 172.21.15.40:0/875673710 >> v2:172.21.15.135:6800/116426 conn(0x7f9d90068960 0x7f9d9006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:17.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.212+0000 7f9db17e6700 1 --2- 172.21.15.40:0/875673710 >> v2:172.21.15.135:6800/116426 conn(0x7f9d90068960 0x7f9d9006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:17.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.212+0000 7f9db17e6700 10 monclient: get_auth_request con 0x7f9d90068960 auth_method 0 2023-09-30T00:53:17.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.213+0000 7f9da17fa700 10 client.4996.objecter ms_handle_connect 0x7f9d90068960 2023-09-30T00:53:17.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.213+0000 7f9db17e6700 1 --2- 172.21.15.40:0/875673710 >> v2:172.21.15.135:6800/116426 conn(0x7f9d90068960 0x7f9d9006ae10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9da8009d10 tx=0x7f9da80091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:17.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.229+0000 7f9da17fa700 1 -- 172.21.15.40:0/875673710 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9da4056780 con 0x7f9dac0b1460 2023-09-30T00:53:17.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.229+0000 7f9da17fa700 10 client.4996.objecter ms_dispatch 0x7f9dac001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:17.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.229+0000 7f9da17fa700 3 client.4996.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:17.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.229+0000 7f9da17fa700 3 client.4996.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:17.229 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.229+0000 7f9da17fa700 20 client.4996.objecter dump_active .. 0 homeless 2023-09-30T00:53:17.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.238+0000 7f9da17fa700 1 -- 172.21.15.40:0/875673710 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9da40523c0 con 0x7f9dac0b1460 2023-09-30T00:53:17.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.238+0000 7f9da17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:17.239 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.238+0000 7f9da17fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:17.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.465+0000 7f9db3a4a700 1 -- 172.21.15.40:0/875673710 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f9dac0a9190 con 0x7f9d90068960 2023-09-30T00:53:17.466 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.465+0000 7f9da17fa700 1 -- 172.21.15.40:0/875673710 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9dac0a9190 con 0x7f9d90068960 2023-09-30T00:53:17.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.467+0000 7f9db3a4a700 10 librados: watch_flush enter 2023-09-30T00:53:17.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.467+0000 7f9db3a4a700 10 librados: watch_flush exit 2023-09-30T00:53:17.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.467+0000 7f9db3a4a700 20 client.4996.objecter shutdown clearing up homeless session... 2023-09-30T00:53:17.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.467+0000 7f9db3a4a700 10 client.4996.objecter successfully canceled tick 2023-09-30T00:53:17.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 1 -- 172.21.15.40:0/875673710 >> v2:172.21.15.135:6800/116426 conn(0x7f9d90068960 msgr2=0x7f9d9006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:17.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 1 --2- 172.21.15.40:0/875673710 >> v2:172.21.15.135:6800/116426 conn(0x7f9d90068960 0x7f9d9006ae10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9da8009d10 tx=0x7f9da80091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:17.469 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 10 monclient: shutdown 2023-09-30T00:53:17.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:17.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 1 -- 172.21.15.40:0/875673710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dac0b1460 msgr2=0x7f9dac13cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:17.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 1 --2- 172.21.15.40:0/875673710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dac0b1460 0x7f9dac13cd30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9da400cb90 tx=0x7f9da40019a0 comp rx=0 tx=0).stop 2023-09-30T00:53:17.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 1 -- 172.21.15.40:0/875673710 shutdown_connections 2023-09-30T00:53:17.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 1 --2- 172.21.15.40:0/875673710 >> v2:172.21.15.135:6800/116426 conn(0x7f9d90068960 0x7f9d9006ae10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:17.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 1 --2- 172.21.15.40:0/875673710 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f9dac0b1460 0x7f9dac13cd30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:17.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 1 --2- 172.21.15.40:0/875673710 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f9dac0a5fb0 0x7f9dac0a8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:17.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 1 --2- 172.21.15.40:0/875673710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9dac0af410 0x7f9dac0a81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:17.470 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 1 -- 172.21.15.40:0/875673710 >> 172.21.15.40:0/875673710 conn(0x7f9dac0631f0 msgr2=0x7f9dac0635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:17.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 1 -- 172.21.15.40:0/875673710 shutdown_connections 2023-09-30T00:53:17.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 1 -- 172.21.15.40:0/875673710 wait complete. 2023-09-30T00:53:17.471 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.468+0000 7f9db3a4a700 1 librados: shutdown 2023-09-30T00:53:17.471 INFO:teuthology.orchestra.run.smithi040.stderr:instructed osd(s) 7 to deep-scrub 2023-09-30T00:53:17.481 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:53:17.618 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.618+0000 7f7e62f2f700 10 monclient: get_monmap_and_config 2023-09-30T00:53:17.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.618+0000 7f7e62f2f700 10 monclient: build_initial_monmap 2023-09-30T00:53:17.619 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.618+0000 7f7e62f2f700 10 monclient: monmap: 2023-09-30T00:53:17.619 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:17.619 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:17.619 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:17.619486+0000 2023-09-30T00:53:17.619 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:17.619486+0000 2023-09-30T00:53:17.619 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:17.619 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:17.619 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:17.619 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:17.620 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:17.620 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:17.620 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:17.620 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:17.621 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.619+0000 7f7e62f2f700 1 Processor -- start 2023-09-30T00:53:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.619+0000 7f7e62f2f700 1 -- start start 2023-09-30T00:53:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.619+0000 7f7e62f2f700 10 monclient: init 2023-09-30T00:53:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.620+0000 7f7e62f2f700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.620+0000 7f7e62f2f700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-09-30T00:53:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.620+0000 7f7e62f2f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e5c0b1460 0x7f7e5c0b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.620+0000 7f7e62f2f700 10 monclient(hunting): picked mon.noname-b con 0x7f7e5c0b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.620+0000 7f7e62f2f700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7e5c0a5fb0 0x7f7e5c141730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.620+0000 7f7e62f2f700 10 monclient(hunting): picked mon.noname-c con 0x7f7e5c0a5fb0 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.620+0000 7f7e62f2f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e5c0af410 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.620+0000 7f7e62f2f700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e5c0a71f0 con 0x7f7e5c0af410 2023-09-30T00:53:17.622 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.620+0000 7f7e62f2f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.620+0000 7f7e62f2f700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e5c0a74d0 con 0x7f7e5c0a5fb0 2023-09-30T00:53:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.620+0000 7f7e62f2f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.620+0000 7f7e62f2f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0a63c0 con 0x7f7e5c0b1460 2023-09-30T00:53:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.620+0000 7f7e62f2f700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.620+0000 7f7e62f2f700 10 monclient(hunting): authenticate will time out at 5658.921402s 2023-09-30T00:53:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.621+0000 7f7e5bfff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7e5c0a5fb0 0x7f7e5c141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.621+0000 7f7e5bfff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7f7e5c0a5fb0 0x7f7e5c141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:56650/0 (socket says 172.21.15.40:56650) 2023-09-30T00:53:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.621+0000 7f7e5bfff700 1 -- 172.21.15.40:0/2090636658 learned_addr learned my addr 172.21.15.40:0/2090636658 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.621+0000 7f7e5bfff700 10 monclient(hunting): get_auth_request con 0x7f7e5c0a5fb0 auth_method 0 2023-09-30T00:53:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.621+0000 7f7e5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.621+0000 7f7e5bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:17.623 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.621+0000 7f7e5bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:17.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.622+0000 7f7e60ccb700 1 --2- 172.21.15.40:0/2090636658 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e5c0b1460 0x7f7e5c0b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:17.624 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.622+0000 7f7e5bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:17.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.622+0000 7f7e5bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:17.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.622+0000 7f7e5bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:17.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.622+0000 7f7e60ccb700 10 monclient(hunting): get_auth_request con 0x7f7e5c0b1460 auth_method 0 2023-09-30T00:53:17.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.622+0000 7f7e60ccb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:17.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.622+0000 7f7e60ccb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:17.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.622+0000 7f7e60ccb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:17.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.623+0000 7f7e60ccb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:17.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.623+0000 7f7e60ccb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:17.625 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.623+0000 7f7e60ccb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:17.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.624+0000 7f7e5bfff700 10 monclient(hunting): handle_auth_done global_id 4999 payload 274 2023-09-30T00:53:17.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.624+0000 7f7e5bfff700 1 -- 172.21.15.40:0/2090636658 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e5c0b1460 msgr2=0x7f7e5c0b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:17.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.624+0000 7f7e5bfff700 1 --2- 172.21.15.40:0/2090636658 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e5c0b1460 0x7f7e5c0b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:17.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.624+0000 7f7e5bfff700 1 -- 172.21.15.40:0/2090636658 >> v1:172.21.15.135:6789/0 conn(0x7f7e5c0af410 legacy=0x7f7e5c0af890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:17.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.624+0000 7f7e5bfff700 10 monclient: _finish_hunting 0 2023-09-30T00:53:17.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.624+0000 7f7e5bfff700 1 monclient: found mon.noname-c 2023-09-30T00:53:17.626 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.624+0000 7f7e5bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:17.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.624+0000 7f7e5bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.135:3300/0 2023-09-30T00:53:17.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.624+0000 7f7e5bfff700 1 -- 172.21.15.40:0/2090636658 --> v2:172.21.15.135:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5c0a56d0 con 0x7f7e5c0a5fb0 2023-09-30T00:53:17.627 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.627+0000 7f7e5bfff700 1 --2- 172.21.15.40:0/2090636658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e5c0a5fb0 0x7f7e5c141730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7e480099b0 tx=0x7f7e48009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68ec80f05a59c93e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:17.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.630+0000 7f7e5b7fe700 1 -- 172.21.15.40:0/2090636658 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48009ea0 con 0x7f7e5c0a5fb0 2023-09-30T00:53:17.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.630+0000 7f7e5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:17.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.630+0000 7f7e5b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:53:17.631 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.630+0000 7f7e5b7fe700 10 monclient: dump: 2023-09-30T00:53:17.631 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:17.631 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:17.631 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:17.631 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:17.632 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:17.632 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:17.632 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:17.632 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:17.632 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:17.632 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:17.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.630+0000 7f7e5b7fe700 10 monclient: _finish_auth 0 2023-09-30T00:53:17.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.630+0000 7f7e5b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:17.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.630+0000 7f7e62f2f700 5 monclient: authenticate success, global_id 4999 2023-09-30T00:53:17.632 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.630+0000 7f7e62f2f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:53:17.633 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.632+0000 7f7e5b7fe700 1 -- 172.21.15.40:0/2090636658 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4800b860 con 0x7f7e5c0a5fb0 2023-09-30T00:53:17.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.632+0000 7f7e5b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:17.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.632+0000 7f7e5b7fe700 1 -- 172.21.15.40:0/2090636658 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4800bc00 con 0x7f7e5c0a5fb0 2023-09-30T00:53:17.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.632+0000 7f7e5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:17.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.632+0000 7f7e5b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:17.634 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.632+0000 7f7e5b7fe700 10 monclient: dump: 2023-09-30T00:53:17.634 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:17.634 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:17.634 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:17.634 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:17.634 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:17.634 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:17.635 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:17.635 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:17.635 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:17.635 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:17.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.632+0000 7f7e62f2f700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:17.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.632+0000 7f7e62f2f700 4 set_mon_vals no callback set 2023-09-30T00:53:17.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.633+0000 7f7e62f2f700 10 monclient: shutdown 2023-09-30T00:53:17.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.633+0000 7f7e62f2f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:17.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.633+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2090636658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e5c0a5fb0 msgr2=0x7f7e5c141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:17.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.633+0000 7f7e62f2f700 1 --2- 172.21.15.40:0/2090636658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e5c0a5fb0 0x7f7e5c141730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7e480099b0 tx=0x7f7e48009250 comp rx=0 tx=0).stop 2023-09-30T00:53:17.635 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.633+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2090636658 shutdown_connections 2023-09-30T00:53:17.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.633+0000 7f7e62f2f700 1 --2- 172.21.15.40:0/2090636658 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e5c0b1460 0x7f7e5c0b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:17.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.633+0000 7f7e62f2f700 1 --2- 172.21.15.40:0/2090636658 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e5c0a5fb0 0x7f7e5c141730 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:17.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.633+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2090636658 >> 172.21.15.40:0/2090636658 conn(0x7f7e5c0631f0 msgr2=0x7f7e5c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:17.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.633+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2090636658 shutdown_connections 2023-09-30T00:53:17.636 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.633+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2090636658 wait complete. 2023-09-30T00:53:17.638 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.634+0000 7f7e62f2f700 10 monclient: build_initial_monmap 2023-09-30T00:53:17.639 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.634+0000 7f7e62f2f700 10 monclient: monmap: 2023-09-30T00:53:17.639 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:17.639 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:17.639 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:17.639 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:17.639 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:17.639 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:17.639 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:17.639 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:17.640 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:17.640 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:17.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.634+0000 7f7e62f2f700 1 librados: starting msgr at 2023-09-30T00:53:17.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.634+0000 7f7e62f2f700 1 librados: starting objecter 2023-09-30T00:53:17.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.634+0000 7f7e62f2f700 1 Processor -- start 2023-09-30T00:53:17.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.634+0000 7f7e62f2f700 1 -- start start 2023-09-30T00:53:17.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.634+0000 7f7e62f2f700 1 librados: setting wanted keys 2023-09-30T00:53:17.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.634+0000 7f7e62f2f700 1 librados: calling monclient init 2023-09-30T00:53:17.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.634+0000 7f7e62f2f700 10 monclient: init 2023-09-30T00:53:17.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:17.640 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:53:17.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e5c0a5fb0 0x7f7e5c0a82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:17.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 10 monclient(hunting): picked mon.noname-c con 0x7f7e5c0a5fb0 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:17.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e5c0af410 0x7f7e5c13d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:17.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 10 monclient(hunting): picked mon.noname-a con 0x7f7e5c0af410 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:17.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e5c0b1460 0x7f7e5c13d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:17.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 10 monclient(hunting): picked mon.noname-b con 0x7f7e5c0b1460 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:17.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:17.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0a8f10 con 0x7f7e5c0af410 2023-09-30T00:53:17.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:17.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0a9f20 con 0x7f7e5c0b1460 2023-09-30T00:53:17.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:17.641 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5c0a9180 con 0x7f7e5c0a5fb0 2023-09-30T00:53:17.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:17.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.635+0000 7f7e62f2f700 10 monclient(hunting): authenticate will time out at 5658.936555s 2023-09-30T00:53:17.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e5bfff700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e5c0af410 0x7f7e5c13d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:17.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e614cc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e5c0b1460 0x7f7e5c13d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:17.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e60ccb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e5c0a5fb0 0x7f7e5c0a82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:17.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e614cc700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e5c0b1460 0x7f7e5c13d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:56652/0 (socket says 172.21.15.40:56652) 2023-09-30T00:53:17.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e614cc700 1 -- 172.21.15.40:0/2205907146 learned_addr learned my addr 172.21.15.40:0/2205907146 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:17.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e614cc700 10 monclient(hunting): get_auth_request con 0x7f7e5c0b1460 auth_method 0 2023-09-30T00:53:17.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e614cc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:17.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e614cc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:17.642 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e614cc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:17.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e60ccb700 10 monclient(hunting): get_auth_request con 0x7f7e5c0a5fb0 auth_method 0 2023-09-30T00:53:17.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e60ccb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:17.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e60ccb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:17.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e60ccb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:17.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e5bfff700 10 monclient(hunting): get_auth_request con 0x7f7e5c0af410 auth_method 0 2023-09-30T00:53:17.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:17.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e5bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:17.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.636+0000 7f7e5bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:17.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.637+0000 7f7e614cc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:17.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.637+0000 7f7e614cc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:17.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.637+0000 7f7e614cc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:17.643 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.637+0000 7f7e60ccb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:17.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.637+0000 7f7e60ccb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:17.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.637+0000 7f7e60ccb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:17.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.637+0000 7f7e5bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:17.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.637+0000 7f7e5bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:17.644 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.637+0000 7f7e5bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:17.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.640+0000 7f7e614cc700 10 monclient(hunting): handle_auth_done global_id 5002 payload 824 2023-09-30T00:53:17.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.640+0000 7f7e614cc700 1 -- 172.21.15.40:0/2205907146 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e5c0a5fb0 msgr2=0x7f7e5c0a82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:17.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.640+0000 7f7e614cc700 1 --2- 172.21.15.40:0/2205907146 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e5c0a5fb0 0x7f7e5c0a82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:17.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.640+0000 7f7e614cc700 1 -- 172.21.15.40:0/2205907146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e5c0af410 msgr2=0x7f7e5c13d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:17.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.640+0000 7f7e614cc700 1 --2- 172.21.15.40:0/2205907146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e5c0af410 0x7f7e5c13d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:17.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.640+0000 7f7e614cc700 10 monclient: _finish_hunting 0 2023-09-30T00:53:17.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.640+0000 7f7e614cc700 1 monclient: found mon.noname-b 2023-09-30T00:53:17.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.640+0000 7f7e614cc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:17.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.640+0000 7f7e614cc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:17.645 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.640+0000 7f7e614cc700 1 -- 172.21.15.40:0/2205907146 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e480094f0 con 0x7f7e5c0b1460 2023-09-30T00:53:17.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.642+0000 7f7e614cc700 1 --2- 172.21.15.40:0/2205907146 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e5c0b1460 0x7f7e5c13d5b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7e5400c8e0 tx=0x7f7e54001710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:17.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.642+0000 7f7e58ff9700 10 client.?.objecter ms_handle_connect 0x7f7e5c0b1460 2023-09-30T00:53:17.646 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.642+0000 7f7e58ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:17.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e58ff9700 1 -- 172.21.15.40:0/2205907146 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e54012630 con 0x7f7e5c0b1460 2023-09-30T00:53:17.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:17.647 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e58ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:17.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e58ff9700 10 monclient: dump: 2023-09-30T00:53:17.648 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:17.648 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:17.648 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:17.648 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:17.648 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:17.648 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:17.648 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:17.648 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:17.648 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:17.648 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:17.648 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e58ff9700 10 monclient: _finish_auth 0 2023-09-30T00:53:17.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e58ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:17.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e58ff9700 1 -- 172.21.15.40:0/2205907146 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e54012790 con 0x7f7e5c0b1460 2023-09-30T00:53:17.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e58ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:17.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e58ff9700 1 -- 172.21.15.40:0/2205907146 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e54012b50 con 0x7f7e5c0b1460 2023-09-30T00:53:17.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:17.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e58ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:17.649 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e58ff9700 10 monclient: dump: 2023-09-30T00:53:17.649 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:17.649 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:17.649 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:17.649 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:17.649 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:17.650 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:17.650 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:17.650 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:17.650 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:17.650 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:17.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e62f2f700 5 monclient: authenticate success, global_id 5002 2023-09-30T00:53:17.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e62f2f700 10 monclient: _renew_subs 2023-09-30T00:53:17.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e62f2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:17.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2205907146 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5c0aa400 con 0x7f7e5c0b1460 2023-09-30T00:53:17.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e5a7fc700 4 set_mon_vals no callback set 2023-09-30T00:53:17.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e62f2f700 10 client.5002.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:17.650 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e62f2f700 10 monclient: _renew_subs 2023-09-30T00:53:17.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e62f2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:17.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2205907146 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5c13ecd0 con 0x7f7e5c0b1460 2023-09-30T00:53:17.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.646+0000 7f7e62f2f700 1 librados: init done 2023-09-30T00:53:17.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.647+0000 7f7e62f2f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:17.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.647+0000 7f7e62f2f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:17.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.647+0000 7f7e62f2f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:17.651 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.647+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2205907146 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5c0aa180 con 0x7f7e5c0b1460 2023-09-30T00:53:17.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.666+0000 7f7e58ff9700 1 -- 172.21.15.40:0/2205907146 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7e54024420 con 0x7f7e5c0b1460 2023-09-30T00:53:17.666 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.666+0000 7f7e58ff9700 1 --2- 172.21.15.40:0/2205907146 >> v2:172.21.15.135:6800/116426 conn(0x7f7e40068960 0x7f7e4006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:17.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.666+0000 7f7e60ccb700 1 --2- 172.21.15.40:0/2205907146 >> v2:172.21.15.135:6800/116426 conn(0x7f7e40068960 0x7f7e4006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:17.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.666+0000 7f7e60ccb700 10 monclient: get_auth_request con 0x7f7e40068960 auth_method 0 2023-09-30T00:53:17.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.667+0000 7f7e58ff9700 10 client.5002.objecter ms_handle_connect 0x7f7e40068960 2023-09-30T00:53:17.667 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.667+0000 7f7e60ccb700 1 --2- 172.21.15.40:0/2205907146 >> v2:172.21.15.135:6800/116426 conn(0x7f7e40068960 0x7f7e4006ae10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7e50009d10 tx=0x7f7e500091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:17.900 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.900+0000 7f7e58ff9700 1 -- 172.21.15.40:0/2205907146 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7e54056470 con 0x7f7e5c0b1460 2023-09-30T00:53:17.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.900+0000 7f7e58ff9700 10 client.5002.objecter ms_dispatch 0x7f7e5c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:17.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.900+0000 7f7e58ff9700 3 client.5002.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:17.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.900+0000 7f7e58ff9700 3 client.5002.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:17.901 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.900+0000 7f7e58ff9700 20 client.5002.objecter dump_active .. 0 homeless 2023-09-30T00:53:17.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.910+0000 7f7e58ff9700 1 -- 172.21.15.40:0/2205907146 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7e540520b0 con 0x7f7e5c0b1460 2023-09-30T00:53:17.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.910+0000 7f7e58ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:17.910 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:17.910+0000 7f7e58ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:18.114 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.113+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2205907146 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7e5c064a00 con 0x7f7e40068960 2023-09-30T00:53:18.117 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.117+0000 7f7e58ff9700 1 -- 172.21.15.40:0/2205907146 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70137 (secure 0 0 0) 0x7f7e5c064a00 con 0x7f7e40068960 2023-09-30T00:53:18.118 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:53:18.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 10 librados: watch_flush enter 2023-09-30T00:53:18.120 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 10 librados: watch_flush exit 2023-09-30T00:53:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 20 client.5002.objecter shutdown clearing up homeless session... 2023-09-30T00:53:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 10 client.5002.objecter successfully canceled tick 2023-09-30T00:53:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2205907146 >> v2:172.21.15.135:6800/116426 conn(0x7f7e40068960 msgr2=0x7f7e4006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 1 --2- 172.21.15.40:0/2205907146 >> v2:172.21.15.135:6800/116426 conn(0x7f7e40068960 0x7f7e4006ae10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7e50009d10 tx=0x7f7e500091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 10 monclient: shutdown 2023-09-30T00:53:18.121 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:18.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2205907146 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e5c0b1460 msgr2=0x7f7e5c13d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:18.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 1 --2- 172.21.15.40:0/2205907146 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e5c0b1460 0x7f7e5c13d5b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7e5400c8e0 tx=0x7f7e54001710 comp rx=0 tx=0).stop 2023-09-30T00:53:18.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2205907146 shutdown_connections 2023-09-30T00:53:18.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 1 --2- 172.21.15.40:0/2205907146 >> v2:172.21.15.135:6800/116426 conn(0x7f7e40068960 0x7f7e4006ae10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:18.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 1 --2- 172.21.15.40:0/2205907146 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f7e5c0b1460 0x7f7e5c13d5b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:18.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 1 --2- 172.21.15.40:0/2205907146 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f7e5c0a5fb0 0x7f7e5c0a82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:18.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 1 --2- 172.21.15.40:0/2205907146 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e5c0af410 0x7f7e5c13d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:18.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2205907146 >> 172.21.15.40:0/2205907146 conn(0x7f7e5c0631f0 msgr2=0x7f7e5c144b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:18.122 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2205907146 shutdown_connections 2023-09-30T00:53:18.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 1 -- 172.21.15.40:0/2205907146 wait complete. 2023-09-30T00:53:18.123 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:18.119+0000 7f7e62f2f700 1 librados: shutdown 2023-09-30T00:53:18.123 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-09-30T00:53:18.133 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2023-09-30T00:53:17.940873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43088,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931632,"statfs":{"total":773094113280,"available":773049991168,"internally_reserved":0,"allocated":2506752,"data_stored":1383352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":68,"apply_latency_ms":68,"commit_latency_ns":68000000,"apply_latency_ns":68000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002274"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.987611+0000","last_change":"2023-09-30T00:47:54.691712+0000","last_active":"2023-09-30T00:47:55.987611+0000","last_peered":"2023-09-30T00:47:55.987611+0000","last_clean":"2023-09-30T00:47:55.987611+0000","last_became_active":"2023-09-30T00:47:54.672127+0000","last_became_peered":"2023-09-30T00:47:54.672127+0000","last_unstale":"2023-09-30T00:47:55.987611+0000","last_undegraded":"2023-09-30T00:47:55.987611+0000","last_fullsized":"2023-09-30T00:47:55.987611+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:47:56.176189+0000","last_change":"2023-09-30T00:47:54.749019+0000","last_active":"2023-09-30T00:47:56.176189+0000","last_peered":"2023-09-30T00:47:56.176189+0000","last_clean":"2023-09-30T00:47:56.176189+0000","last_became_active":"2023-09-30T00:47:54.702773+0000","last_became_peered":"2023-09-30T00:47:54.702773+0000","last_unstale":"2023-09-30T00:47:56.176189+0000","last_undegraded":"2023-09-30T00:47:56.176189+0000","last_fullsized":"2023-09-30T00:47:56.176189+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.981288+0000","last_change":"2023-09-30T00:47:54.157108+0000","last_active":"2023-09-30T00:47:55.981288+0000","last_peered":"2023-09-30T00:47:55.981288+0000","last_clean":"2023-09-30T00:47:55.981288+0000","last_became_active":"2023-09-30T00:47:54.139464+0000","last_became_peered":"2023-09-30T00:47:54.139464+0000","last_unstale":"2023-09-30T00:47:55.981288+0000","last_undegraded":"2023-09-30T00:47:55.981288+0000","last_fullsized":"2023-09-30T00:47:55.981288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.971306+0000","last_change":"2023-09-30T00:47:53.993668+0000","last_active":"2023-09-30T00:47:55.971306+0000","last_peered":"2023-09-30T00:47:55.971306+0000","last_clean":"2023-09-30T00:47:55.971306+0000","last_became_active":"2023-09-30T00:47:53.951882+0000","last_became_peered":"2023-09-30T00:47:53.951882+0000","last_unstale":"2023-09-30T00:47:55.971306+0000","last_undegraded":"2023-09-30T00:47:55.971306+0000","last_fullsized":"2023-09-30T00:47:55.971306+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'79","reported_seq":127,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:53:13.271839+0000","last_change":"2023-09-30T00:53:13.262553+0000","last_active":"2023-09-30T00:53:13.271839+0000","last_peered":"2023-09-30T00:53:13.271839+0000","last_clean":"2023-09-30T00:53:13.271839+0000","last_became_active":"2023-09-30T00:47:46.406306+0000","last_became_peered":"2023-09-30T00:47:46.406306+0000","last_unstale":"2023-09-30T00:53:13.271839+0000","last_undegraded":"2023-09-30T00:53:13.271839+0000","last_fullsized":"2023-09-30T00:53:13.271839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'79","last_scrub_stamp":"2023-09-30T00:53:13.250386+0000","last_deep_scrub":"25'79","last_deep_scrub_stamp":"2023-09-30T00:53:13.250386+0000","last_clean_scrub_stamp":"2023-09-30T00:53:13.250386+0000","objects_scrubbed":2,"log_size":79,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:21:58.427216+0000","scrub_duration":0.73232861699999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.978823+0000","last_change":"2023-09-30T00:47:54.729272+0000","last_active":"2023-09-30T00:47:55.978823+0000","last_peered":"2023-09-30T00:47:55.978823+0000","last_clean":"2023-09-30T00:47:55.978823+0000","last_became_active":"2023-09-30T00:47:54.710752+0000","last_became_peered":"2023-09-30T00:47:54.710752+0000","last_unstale":"2023-09-30T00:47:55.978823+0000","last_undegraded":"2023-09-30T00:47:55.978823+0000","last_fullsized":"2023-09-30T00:47:55.978823+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.239361+0000","last_change":"2023-09-30T00:47:53.773125+0000","last_active":"2023-09-30T00:47:55.239361+0000","last_peered":"2023-09-30T00:47:55.239361+0000","last_clean":"2023-09-30T00:47:55.239361+0000","last_became_active":"2023-09-30T00:47:53.761460+0000","last_became_peered":"2023-09-30T00:47:53.761460+0000","last_unstale":"2023-09-30T00:47:55.239361+0000","last_undegraded":"2023-09-30T00:47:55.239361+0000","last_fullsized":"2023-09-30T00:47:55.239361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:08:55.538707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.975237+0000","last_change":"2023-09-30T00:47:54.294197+0000","last_active":"2023-09-30T00:47:55.975237+0000","last_peered":"2023-09-30T00:47:55.975237+0000","last_clean":"2023-09-30T00:47:55.975237+0000","last_became_active":"2023-09-30T00:47:54.279706+0000","last_became_peered":"2023-09-30T00:47:54.279706+0000","last_unstale":"2023-09-30T00:47:55.975237+0000","last_undegraded":"2023-09-30T00:47:55.975237+0000","last_fullsized":"2023-09-30T00:47:55.975237+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T00:47:55.240181+0000","last_change":"2023-09-30T00:47:54.352713+0000","last_active":"2023-09-30T00:47:55.240181+0000","last_peered":"2023-09-30T00:47:55.240181+0000","last_clean":"2023-09-30T00:47:55.240181+0000","last_became_active":"2023-09-30T00:47:54.305507+0000","last_became_peered":"2023-09-30T00:47:54.305507+0000","last_unstale":"2023-09-30T00:47:55.240181+0000","last_undegraded":"2023-09-30T00:47:55.240181+0000","last_fullsized":"2023-09-30T00:47:55.240181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:47:51.911887+0000","last_clean_scrub_stamp":"2023-09-30T00:47:51.911887+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:21:08.873800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345991,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":13.013999999999999,"5min":12.769,"15min":13.343},"min":{"1min":7.7699999999999996,"5min":6.6319999999999997,"15min":6.6319999999999997},"max":{"1min":20.431999999999999,"5min":26.166,"15min":26.166},"last":15.367000000000001},{"interface":"front","average":{"1min":10.711,"5min":11.778,"15min":13.131},"min":{"1min":6.8440000000000003,"5min":6.8440000000000003,"15min":6.8440000000000003},"max":{"1min":15.452,"5min":32.808,"15min":32.808},"last":9.1600000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":12.897,"5min":13.236000000000001,"15min":14.891999999999999},"min":{"1min":9.0199999999999996,"5min":7.8250000000000002,"15min":7.8250000000000002},"max":{"1min":21.581,"5min":31.925999999999998,"15min":31.925999999999998},"last":48.212000000000003},{"interface":"front","average":{"1min":15.115,"5min":16.175999999999998,"15min":17.780000000000001},"min":{"1min":9.3369999999999997,"5min":8.4629999999999992,"15min":8.4629999999999992},"max":{"1min":23.300000000000001,"5min":42.737000000000002,"15min":42.737000000000002},"last":49.363}]},{"osd":2,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":14.624000000000001,"5min":16.120999999999999,"15min":17.459},"min":{"1min":8.9489999999999998,"5min":8.3469999999999995,"15min":8.3469999999999995},"max":{"1min":22.853000000000002,"5min":36.436999999999998,"15min":36.436999999999998},"last":10.798},{"interface":"front","average":{"1min":18.524999999999999,"5min":19.745999999999999,"15min":25.524999999999999},"min":{"1min":11.426,"5min":10.092000000000001,"15min":10.092000000000001},"max":{"1min":22.728999999999999,"5min":164.05600000000001,"15min":164.05600000000001},"last":16.789999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":15.416,"5min":17.084,"15min":17.538},"min":{"1min":10.166,"5min":8.9570000000000007,"15min":8.9570000000000007},"max":{"1min":21.091999999999999,"5min":36.740000000000002,"15min":36.740000000000002},"last":14.73},{"interface":"front","average":{"1min":13.753,"5min":15.188000000000001,"15min":17.27},"min":{"1min":8.1069999999999993,"5min":7.4329999999999998,"15min":7.4329999999999998},"max":{"1min":18.597999999999999,"5min":39.865000000000002,"15min":39.865000000000002},"last":19.966999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":18.372,"5min":19.262,"15min":21.207000000000001},"min":{"1min":12.487,"5min":10.294,"15min":10.294},"max":{"1min":22.335999999999999,"5min":47.259,"15min":47.259},"last":22.411000000000001},{"interface":"front","average":{"1min":18.518000000000001,"5min":19.602,"15min":21.158000000000001},"min":{"1min":10.493,"5min":9.7059999999999995,"15min":9.7059999999999995},"max":{"1min":23.382999999999999,"5min":46.417999999999999,"15min":46.417999999999999},"last":18.579000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":20.271000000000001,"5min":20.856999999999999,"15min":22.073},"min":{"1min":13.49,"5min":11.042999999999999,"15min":11.042999999999999},"max":{"1min":25.059999999999999,"5min":49.011000000000003,"15min":49.011000000000003},"last":20.75},{"interface":"front","average":{"1min":16.920999999999999,"5min":17.010999999999999,"15min":18.483000000000001},"min":{"1min":8.9130000000000003,"5min":8.9130000000000003,"15min":8.9130000000000003},"max":{"1min":24.98,"5min":44.369,"15min":44.369},"last":19.266999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":19.391999999999999,"15min":21.218},"min":{"1min":13.249000000000001,"5min":12.590999999999999,"15min":12.590999999999999},"max":{"1min":25.725000000000001,"5min":39.171999999999997,"15min":39.171999999999997},"last":24.861000000000001},{"interface":"front","average":{"1min":19.501999999999999,"5min":21.446999999999999,"15min":23.422000000000001},"min":{"1min":13.321999999999999,"5min":11.161,"15min":11.161},"max":{"1min":24.439,"5min":41.865000000000002,"15min":41.865000000000002},"last":23.831}]}]},{"osd":2,"up_from":20,"seq":85899345990,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":12.666,"5min":12.167,"15min":12.167},"min":{"1min":6.9450000000000003,"5min":6.6920000000000002,"15min":6.6920000000000002},"max":{"1min":20.530999999999999,"5min":22.748999999999999,"15min":22.748999999999999},"last":12.476000000000001},{"interface":"front","average":{"1min":12.608000000000001,"5min":13.143000000000001,"15min":14.31},"min":{"1min":9.4100000000000001,"5min":6.319,"15min":6.319},"max":{"1min":21.957000000000001,"5min":31.885000000000002,"15min":31.885000000000002},"last":12.154}]},{"osd":1,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":15.48,"5min":17.213000000000001,"15min":22.105},"min":{"1min":9.6579999999999995,"5min":8.718,"15min":8.718},"max":{"1min":21.058,"5min":125.67700000000001,"15min":125.67700000000001},"last":14.32},{"interface":"front","average":{"1min":16.795999999999999,"5min":17.684999999999999,"15min":24.204999999999998},"min":{"1min":10.210000000000001,"5min":8.0370000000000008,"15min":8.0370000000000008},"max":{"1min":24.420999999999999,"5min":156.821,"15min":156.821},"last":7.8399999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":10.301,"5min":18.065999999999999,"15min":34.295999999999999},"min":{"1min":7.6210000000000004,"5min":7.6210000000000004,"15min":7.6210000000000004},"max":{"1min":14.249000000000001,"5min":513.32600000000002,"15min":513.32600000000002},"last":27.527000000000001},{"interface":"front","average":{"1min":17.056999999999999,"5min":24.167999999999999,"15min":42.537999999999997},"min":{"1min":11.009,"5min":10.308,"15min":10.308},"max":{"1min":23.140999999999998,"5min":598.48900000000003,"15min":598.48900000000003},"last":19.359000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":14.541,"5min":16.052,"15min":17.638999999999999},"min":{"1min":9.1530000000000005,"5min":8.8859999999999992,"15min":8.8859999999999992},"max":{"1min":24.239999999999998,"5min":31.702000000000002,"15min":31.702000000000002},"last":15.308},{"interface":"front","average":{"1min":16.594000000000001,"5min":16.542000000000002,"15min":17.376999999999999},"min":{"1min":8.6750000000000007,"5min":8.6750000000000007,"15min":8.6750000000000007},"max":{"1min":23.922999999999998,"5min":37.073999999999998,"15min":37.073999999999998},"last":16.100999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":19.826000000000001,"5min":20.279,"15min":21.254000000000001},"min":{"1min":13.452999999999999,"5min":9.1440000000000001,"15min":9.1440000000000001},"max":{"1min":25.805,"5min":38.981000000000002,"15min":38.981000000000002},"last":20.388999999999999},{"interface":"front","average":{"1min":18.370999999999999,"5min":20.248000000000001,"15min":23.968},"min":{"1min":13.335000000000001,"5min":9.8949999999999996,"15min":9.8949999999999996},"max":{"1min":24.638999999999999,"5min":74.510000000000005,"15min":74.510000000000005},"last":20.841999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":18.266999999999999,"5min":17.795000000000002,"15min":19.167000000000002},"min":{"1min":13.561,"5min":7.6150000000000002,"15min":7.6150000000000002},"max":{"1min":24.135999999999999,"5min":40.875999999999998,"15min":40.875999999999998},"last":26.190000000000001},{"interface":"front","average":{"1min":22.277000000000001,"5min":21.561,"15min":23.045999999999999},"min":{"1min":15.621,"5min":9.5809999999999995,"15min":9.5809999999999995},"max":{"1min":26.98,"5min":42.539000000000001,"15min":42.539000000000001},"last":21.757000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":20.937000000000001,"5min":21.863,"15min":24.445},"min":{"1min":14.688000000000001,"5min":12.295999999999999,"15min":12.295999999999999},"max":{"1min":26.420999999999999,"5min":76.484999999999999,"15min":76.484999999999999},"last":25.132000000000001},{"interface":"front","average":{"1min":19.401,"5min":20.391999999999999,"15min":22.324000000000002},"min":{"1min":14.641999999999999,"5min":10.223000000000001,"15min":10.223000000000001},"max":{"1min":28.007999999999999,"5min":55.265999999999998,"15min":55.265999999999998},"last":22.151}]}]},{"osd":5,"up_from":15,"seq":64424509511,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5920,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365920,"statfs":{"total":96636764160,"available":96630702080,"internally_reserved":0,"allocated":614400,"data_stored":471299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":10.923,"5min":10.705,"15min":11.093},"min":{"1min":7.1210000000000004,"5min":5.6130000000000004,"15min":5.6130000000000004},"max":{"1min":18.591999999999999,"5min":23.507000000000001,"15min":23.507000000000001},"last":9.7949999999999999},{"interface":"front","average":{"1min":11.551,"5min":10.593999999999999,"15min":10.163},"min":{"1min":7.1470000000000002,"5min":5.2130000000000001,"15min":5.2130000000000001},"max":{"1min":20.713000000000001,"5min":27.425000000000001,"15min":27.425000000000001},"last":6.5750000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.073,"5min":12.210000000000001,"15min":12.234},"min":{"1min":6.8369999999999997,"5min":6.6619999999999999,"15min":6.6619999999999999},"max":{"1min":15.596,"5min":31.512,"15min":31.512},"last":10.493},{"interface":"front","average":{"1min":16.539000000000001,"5min":19.786999999999999,"15min":26.213000000000001},"min":{"1min":11.753,"5min":10.006,"15min":10.006},"max":{"1min":21.158000000000001,"5min":232.38800000000001,"15min":232.38800000000001},"last":11.679}]},{"osd":2,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":9.3290000000000006,"5min":10.186,"15min":9.9290000000000003},"min":{"1min":7.0999999999999996,"5min":6.9909999999999997,"15min":6.9909999999999997},"max":{"1min":15.125999999999999,"5min":22.271000000000001,"15min":22.271000000000001},"last":18.016999999999999},{"interface":"front","average":{"1min":16.318999999999999,"5min":16.466000000000001,"15min":17.452999999999999},"min":{"1min":12.247999999999999,"5min":8.3810000000000002,"15min":8.3810000000000002},"max":{"1min":19.457000000000001,"5min":30.27,"15min":30.27},"last":16.163}]},{"osd":3,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.276,"5min":13.022,"15min":14.42},"min":{"1min":7.1829999999999998,"5min":7.1829999999999998,"15min":7.1829999999999998},"max":{"1min":17.137,"5min":42.023000000000003,"15min":42.023000000000003},"last":7.5759999999999996},{"interface":"front","average":{"1min":14.127000000000001,"5min":14.695,"15min":16.593},"min":{"1min":9.5350000000000001,"5min":7.1029999999999998,"15min":7.1029999999999998},"max":{"1min":21.387,"5min":46.786000000000001,"15min":46.786000000000001},"last":11.967000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":15.722,"5min":14.898,"15min":14.827999999999999},"min":{"1min":10.923,"5min":6.7380000000000004,"15min":6.7380000000000004},"max":{"1min":21.215,"5min":23.788,"15min":23.788},"last":15.573},{"interface":"front","average":{"1min":17.538,"5min":18.876000000000001,"15min":22.504000000000001},"min":{"1min":11.625,"5min":9.2300000000000004,"15min":9.2300000000000004},"max":{"1min":20.460000000000001,"5min":112.548,"15min":112.548},"last":12.766999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":13.363,"5min":14.715999999999999,"15min":17.433},"min":{"1min":6.819,"5min":6.819,"15min":6.819},"max":{"1min":22.664000000000001,"5min":106.333,"15min":106.333},"last":11.119},{"interface":"front","average":{"1min":16.849,"5min":16.956,"15min":19.303000000000001},"min":{"1min":9.9529999999999994,"5min":8.2769999999999992,"15min":8.2769999999999992},"max":{"1min":23.702999999999999,"5min":103.483,"15min":103.483},"last":17.239000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":17.23,"5min":21.253,"15min":29.43},"min":{"1min":11.779999999999999,"5min":10.369999999999999,"15min":10.369999999999999},"max":{"1min":22.402999999999999,"5min":253.74799999999999,"15min":253.74799999999999},"last":14.605},{"interface":"front","average":{"1min":17.359999999999999,"5min":20.033999999999999,"15min":30.213999999999999},"min":{"1min":9.1869999999999994,"5min":9.0310000000000006,"15min":9.0310000000000006},"max":{"1min":21.940000000000001,"5min":328.31799999999998,"15min":328.31799999999998},"last":18.925000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509510,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":9.6129999999999995,"5min":10.558,"15min":12.276},"min":{"1min":6.4080000000000004,"5min":4.556,"15min":4.556},"max":{"1min":19.038,"5min":78.191000000000003,"15min":78.191000000000003},"last":9.3770000000000007},{"interface":"front","average":{"1min":11.808,"5min":11.657,"15min":13.117000000000001},"min":{"1min":6.4189999999999996,"5min":5.5439999999999996,"15min":5.5439999999999996},"max":{"1min":21.591999999999999,"5min":73.787000000000006,"15min":73.787000000000006},"last":5.7240000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":11.846,"5min":13.288,"15min":15.319000000000001},"min":{"1min":6.6219999999999999,"5min":6.6219999999999999,"15min":6.6219999999999999},"max":{"1min":19.167999999999999,"5min":79.558000000000007,"15min":79.558000000000007},"last":6.6500000000000004},{"interface":"front","average":{"1min":13.247999999999999,"5min":14.374000000000001,"15min":16.331},"min":{"1min":7.1130000000000004,"5min":7.1130000000000004,"15min":7.1130000000000004},"max":{"1min":18.385999999999999,"5min":82.048000000000002,"15min":82.048000000000002},"last":15.071}]},{"osd":2,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":15.504,"5min":16.736999999999998,"15min":18.332000000000001},"min":{"1min":9.4640000000000004,"5min":8.2889999999999997,"15min":8.2889999999999997},"max":{"1min":20.145,"5min":98.372,"15min":98.372},"last":16.643999999999998},{"interface":"front","average":{"1min":14.375999999999999,"5min":15.202999999999999,"15min":16.692},"min":{"1min":8.0009999999999994,"5min":7.2750000000000004,"15min":7.2750000000000004},"max":{"1min":22.282,"5min":96.753,"15min":96.753},"last":9.8049999999999997}]},{"osd":3,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":16.495999999999999,"5min":17.460000000000001,"15min":19.975999999999999},"min":{"1min":9.0860000000000003,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":20.835000000000001,"5min":100.212,"15min":100.212},"last":17.596},{"interface":"front","average":{"1min":11.829000000000001,"5min":12.417,"15min":14.603},"min":{"1min":6.8029999999999999,"5min":6.7729999999999997,"15min":6.7729999999999997},"max":{"1min":19.068999999999999,"5min":69.975999999999999,"15min":69.975999999999999},"last":10.368}]},{"osd":5,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":16.097000000000001,"5min":14.657999999999999,"15min":18.02},"min":{"1min":7.9880000000000004,"5min":7.3070000000000004,"15min":7.3070000000000004},"max":{"1min":20.745999999999999,"5min":129.88,"15min":129.88},"last":19.242999999999999},{"interface":"front","average":{"1min":15.327999999999999,"5min":18.414000000000001,"15min":21.858000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":22.792999999999999,"5min":101.876,"15min":101.876},"last":11.516}]},{"osd":6,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":19.172000000000001,"5min":21.908999999999999,"15min":28.599},"min":{"1min":14.031000000000001,"5min":11.853999999999999,"15min":11.853999999999999},"max":{"1min":23.716999999999999,"5min":199.161,"15min":199.161},"last":18.423999999999999},{"interface":"front","average":{"1min":13.936999999999999,"5min":17.481000000000002,"15min":24.242999999999999},"min":{"1min":9.1899999999999995,"5min":8.5060000000000002,"15min":8.5060000000000002},"max":{"1min":23.106000000000002,"5min":205.011,"15min":205.011},"last":14.814}]},{"osd":7,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":16.001000000000001,"5min":18.004999999999999,"15min":22.398},"min":{"1min":9.9570000000000007,"5min":8.2110000000000003,"15min":8.2110000000000003},"max":{"1min":22.222000000000001,"5min":140.71000000000001,"15min":140.71000000000001},"last":14.035},{"interface":"front","average":{"1min":16.390999999999998,"5min":17.071999999999999,"15min":19.221},"min":{"1min":10.544,"5min":5.6509999999999998,"15min":5.6509999999999998},"max":{"1min":23.492999999999999,"5min":101.80500000000001,"15min":101.80500000000001},"last":13.414999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542215,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.535,"5min":13.951000000000001,"15min":22.998000000000001},"min":{"1min":6.5380000000000003,"5min":4.5990000000000002,"15min":4.5990000000000002},"max":{"1min":17.355,"5min":350.05700000000002,"15min":350.05700000000002},"last":9.452},{"interface":"front","average":{"1min":9.7520000000000007,"5min":13.462999999999999,"15min":23.047999999999998},"min":{"1min":5.383,"5min":5.383,"15min":5.383},"max":{"1min":18.678000000000001,"5min":352.40899999999999,"15min":352.40899999999999},"last":6.6050000000000004}]},{"osd":1,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":12.308,"5min":18.155999999999999,"15min":33.904000000000003},"min":{"1min":6.367,"5min":5.5700000000000003,"15min":5.5700000000000003},"max":{"1min":19.513000000000002,"5min":332.67399999999998,"15min":332.67399999999998},"last":12.127000000000001},{"interface":"front","average":{"1min":12.640000000000001,"5min":19.193999999999999,"15min":37.317999999999998},"min":{"1min":7.8559999999999999,"5min":7.2629999999999999,"15min":7.2629999999999999},"max":{"1min":18.895,"5min":359.57900000000001,"15min":359.57900000000001},"last":10.287000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.337999999999999,"5min":18.093,"15min":29.870000000000001},"min":{"1min":6.907,"5min":6.593,"15min":6.593},"max":{"1min":22.527999999999999,"5min":420.69299999999998,"15min":420.69299999999998},"last":10.529},{"interface":"front","average":{"1min":16.591999999999999,"5min":17.097000000000001,"15min":26.565000000000001},"min":{"1min":10.023,"5min":6.0990000000000002,"15min":6.0990000000000002},"max":{"1min":19.998000000000001,"5min":421.416,"15min":421.416},"last":11.835000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":13.147,"5min":14.747,"15min":17.933},"min":{"1min":7.4450000000000003,"5min":5.9710000000000001,"15min":5.9710000000000001},"max":{"1min":21.074999999999999,"5min":142.905,"15min":142.905},"last":18.678999999999998},{"interface":"front","average":{"1min":17.030999999999999,"5min":19.399999999999999,"15min":23.738},"min":{"1min":13.157,"5min":11.380000000000001,"15min":11.380000000000001},"max":{"1min":19.614000000000001,"5min":177.86199999999999,"15min":177.86199999999999},"last":16.518000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":14.191000000000001,"5min":15.872999999999999,"15min":19.667999999999999},"min":{"1min":9.0670000000000002,"5min":7.0270000000000001,"15min":7.0270000000000001},"max":{"1min":20.523,"5min":164.02699999999999,"15min":164.02699999999999},"last":13.992000000000001},{"interface":"front","average":{"1min":17.466999999999999,"5min":19.172999999999998,"15min":23.913},"min":{"1min":10.384,"5min":6.7400000000000002,"15min":6.7400000000000002},"max":{"1min":22.350000000000001,"5min":183.18000000000001,"15min":183.18000000000001},"last":14.938000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":15.390000000000001,"15min":19.122},"min":{"1min":7.5229999999999997,"5min":7.101,"15min":7.101},"max":{"1min":17.652000000000001,"5min":163.00700000000001,"15min":163.00700000000001},"last":15.93},{"interface":"front","average":{"1min":15.941000000000001,"5min":17.882000000000001,"15min":22.271999999999998},"min":{"1min":11.317,"5min":7.2770000000000001,"15min":7.2770000000000001},"max":{"1min":21.721,"5min":177.17400000000001,"15min":177.17400000000001},"last":19.481999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":18.608000000000001,"5min":20.460000000000001,"15min":24.350000000000001},"min":{"1min":12.318,"5min":8.8469999999999995,"15min":8.8469999999999995},"max":{"1min":21.919,"5min":179.86500000000001,"15min":179.86500000000001},"last":13.039999999999999},{"interface":"front","average":{"1min":14.218,"5min":17.603999999999999,"15min":21.802},"min":{"1min":7.4619999999999997,"5min":7.4619999999999997,"15min":7.4619999999999997},"max":{"1min":19.574000000000002,"5min":164.58099999999999,"15min":164.58099999999999},"last":17.811}]}]},{"osd":7,"up_from":15,"seq":64424509510,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":9.2799999999999994,"5min":10.220000000000001,"15min":11.925000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":17.898,"5min":52.545999999999999,"15min":52.545999999999999},"last":8.2609999999999992},{"interface":"front","average":{"1min":10.803000000000001,"5min":11.276999999999999,"15min":13.035},"min":{"1min":7.5449999999999999,"5min":6.6180000000000003,"15min":6.6180000000000003},"max":{"1min":17.611000000000001,"5min":60.087000000000003,"15min":60.087000000000003},"last":9.673}]},{"osd":1,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":12.558999999999999,"5min":15.726000000000001,"15min":23.890999999999998},"min":{"1min":8.3209999999999997,"5min":5.5179999999999998,"15min":5.5179999999999998},"max":{"1min":18.693999999999999,"5min":292.69600000000003,"15min":292.69600000000003},"last":13.038},{"interface":"front","average":{"1min":10.73,"5min":17.122,"15min":33.595999999999997},"min":{"1min":7.3499999999999996,"5min":6.9059999999999997,"15min":6.9059999999999997},"max":{"1min":19.689,"5min":571.03399999999999,"15min":571.03399999999999},"last":11.794}]},{"osd":2,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":16.391999999999999,"5min":16.396999999999998,"15min":18.145},"min":{"1min":11.743,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":22.370000000000001,"5min":62.984000000000002,"15min":62.984000000000002},"last":17.303000000000001},{"interface":"front","average":{"1min":13.821999999999999,"5min":14.734,"15min":16.702000000000002},"min":{"1min":8.3650000000000002,"5min":7.875,"15min":7.875},"max":{"1min":21.393000000000001,"5min":59.280999999999999,"15min":59.280999999999999},"last":15.021000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":13.872999999999999,"5min":16.023,"15min":17.783000000000001},"min":{"1min":9.3239999999999998,"5min":7.8949999999999996,"15min":7.8949999999999996},"max":{"1min":21.843,"5min":65.084000000000003,"15min":65.084000000000003},"last":10.759},{"interface":"front","average":{"1min":16.149999999999999,"5min":14.801,"15min":16.591000000000001},"min":{"1min":8.1530000000000005,"5min":6.3650000000000002,"15min":6.3650000000000002},"max":{"1min":21.564,"5min":56.024999999999999,"15min":56.024999999999999},"last":16.385000000000002}]},{"osd":4,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":14.815,"5min":15.505000000000001,"15min":16.719000000000001},"min":{"1min":9.6229999999999993,"5min":6.5369999999999999,"15min":6.5369999999999999},"max":{"1min":20.928000000000001,"5min":39.575000000000003,"15min":39.575000000000003},"last":16.673999999999999},{"interface":"front","average":{"1min":16.652999999999999,"5min":18.010000000000002,"15min":20.782},"min":{"1min":9.0199999999999996,"5min":8.5459999999999994,"15min":8.5459999999999994},"max":{"1min":20.765999999999998,"5min":113.193,"15min":113.193},"last":21.614000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":18.452000000000002,"5min":18.542000000000002,"15min":19.276},"min":{"1min":15.332000000000001,"5min":10.16,"15min":10.16},"max":{"1min":22.913,"5min":49.593000000000004,"15min":49.593000000000004},"last":19.978000000000002},{"interface":"front","average":{"1min":16.960999999999999,"5min":18.082000000000001,"15min":19.195},"min":{"1min":12.218999999999999,"5min":7.6820000000000004,"15min":7.6820000000000004},"max":{"1min":20.646000000000001,"5min":48.204000000000001,"15min":48.204000000000001},"last":20.565000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":18.614999999999998,"5min":19.957999999999998,"15min":23.268000000000001},"min":{"1min":14.460000000000001,"5min":10.625999999999999,"15min":10.625999999999999},"max":{"1min":23.527000000000001,"5min":115.434,"15min":115.434},"last":18.032},{"interface":"front","average":{"1min":17.231999999999999,"5min":16.93,"15min":17.43},"min":{"1min":12.894,"5min":8.0190000000000001,"15min":8.0190000000000001},"max":{"1min":22.411999999999999,"5min":40.805,"15min":40.805},"last":12.445}]}]},{"osd":0,"up_from":20,"seq":85899345991,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":12.696999999999999,"5min":13.567,"15min":14.483000000000001},"min":{"1min":5.0830000000000002,"5min":5.0830000000000002,"15min":5.0830000000000002},"max":{"1min":20.931000000000001,"5min":29.138999999999999,"15min":29.138999999999999},"last":10.696999999999999},{"interface":"front","average":{"1min":11.182,"5min":12.558999999999999,"15min":13.718999999999999},"min":{"1min":6.9989999999999997,"5min":6.1059999999999999,"15min":6.1059999999999999},"max":{"1min":20.055,"5min":35.32,"15min":35.32},"last":11.087}]},{"osd":2,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":9.7750000000000004,"5min":11.563000000000001,"15min":14.34},"min":{"1min":7.3109999999999999,"5min":6.9829999999999997,"15min":6.9829999999999997},"max":{"1min":14.356,"5min":56.628999999999998,"15min":56.628999999999998},"last":11.474},{"interface":"front","average":{"1min":16.048999999999999,"5min":17.128,"15min":19.702999999999999},"min":{"1min":11.364000000000001,"5min":10.457000000000001,"15min":10.457000000000001},"max":{"1min":20.681000000000001,"5min":50.646000000000001,"15min":50.646000000000001},"last":13.191000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:53:06 2023","interfaces":[{"interface":"back","average":{"1min":14.436,"5min":21.872,"15min":41.698},"min":{"1min":9.6940000000000008,"5min":8.4399999999999995,"15min":8.4399999999999995},"max":{"1min":20.186,"5min":616.83500000000004,"15min":616.83500000000004},"last":14.744999999999999},{"interface":"front","average":{"1min":14.864000000000001,"5min":21.122,"15min":36.988999999999997},"min":{"1min":8.2059999999999995,"5min":8.2059999999999995,"15min":8.2059999999999995},"max":{"1min":20.460000000000001,"5min":495.32600000000002,"15min":495.32600000000002},"last":16.981999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":17.096,"5min":17.635999999999999,"15min":19.018999999999998},"min":{"1min":11.279999999999999,"5min":9.875,"15min":9.875},"max":{"1min":23.553999999999998,"5min":36.109000000000002,"15min":36.109000000000002},"last":15.557},{"interface":"front","average":{"1min":15.769,"5min":16.484999999999999,"15min":18.593},"min":{"1min":10.702,"5min":8.4420000000000002,"15min":8.4420000000000002},"max":{"1min":22.722999999999999,"5min":39.177999999999997,"15min":39.177999999999997},"last":16.238}]},{"osd":5,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":18.515000000000001,"5min":19.923999999999999,"15min":22.416},"min":{"1min":12.333,"5min":9.3870000000000005,"15min":9.3870000000000005},"max":{"1min":26.995999999999999,"5min":66.313999999999993,"15min":66.313999999999993},"last":20},{"interface":"front","average":{"1min":20.390999999999998,"5min":22.050999999999998,"15min":23.927},"min":{"1min":12.085000000000001,"5min":12.085000000000001,"15min":12.085000000000001},"max":{"1min":25.503,"5min":41.634,"15min":41.634},"last":21.303000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":14.702999999999999,"5min":17.117999999999999,"15min":22.646000000000001},"min":{"1min":10.412000000000001,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":21.728000000000002,"5min":54.033999999999999,"15min":54.033999999999999},"last":19.210999999999999},{"interface":"front","average":{"1min":20.824999999999999,"5min":22.52,"15min":24.59},"min":{"1min":16.77,"5min":11.055999999999999,"15min":11.055999999999999},"max":{"1min":24.806000000000001,"5min":69.688000000000002,"15min":69.688000000000002},"last":23.273}]},{"osd":7,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":22.016999999999999,"5min":23.597999999999999,"15min":26.277000000000001},"min":{"1min":16.298999999999999,"5min":14.875,"15min":14.875},"max":{"1min":27.872,"5min":49.347999999999999,"15min":49.347999999999999},"last":22.286999999999999},{"interface":"front","average":{"1min":16.75,"5min":18.710999999999999,"15min":21.469000000000001},"min":{"1min":10.622,"5min":10.489000000000001,"15min":10.489000000000001},"max":{"1min":24.265000000000001,"5min":54.734999999999999,"15min":54.734999999999999},"last":13.978}]}]},{"osd":1,"up_from":19,"seq":81604378694,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5856,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365984,"statfs":{"total":96636764160,"available":96630767616,"internally_reserved":0,"allocated":614400,"data_stored":471299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":13.113,"15min":16.02},"min":{"1min":7.4809999999999999,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":13.616,"5min":91.582999999999998,"15min":91.582999999999998},"last":18.588000000000001},{"interface":"front","average":{"1min":12.308,"5min":13.007999999999999,"15min":14.891999999999999},"min":{"1min":8.1780000000000008,"5min":7.2889999999999997,"15min":7.2889999999999997},"max":{"1min":26.073,"5min":75.853999999999999,"15min":75.853999999999999},"last":10.050000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":12.657999999999999,"5min":14.127000000000001,"15min":18.731000000000002},"min":{"1min":8.2639999999999993,"5min":8.2639999999999993,"15min":8.2639999999999993},"max":{"1min":20.702000000000002,"5min":142.41999999999999,"15min":142.41999999999999},"last":10.747},{"interface":"front","average":{"1min":15.305,"5min":16.303999999999998,"15min":19.814},"min":{"1min":9.952,"5min":8.0749999999999993,"15min":8.0749999999999993},"max":{"1min":23.823,"5min":133.988,"15min":133.988},"last":11.446999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":14.644,"5min":17.609000000000002,"15min":23.707000000000001},"min":{"1min":8.9009999999999998,"5min":8.4019999999999992,"15min":8.4019999999999992},"max":{"1min":21.539000000000001,"5min":170.785,"15min":170.785},"last":20.785},{"interface":"front","average":{"1min":16.152000000000001,"5min":19.57,"15min":27.786000000000001},"min":{"1min":9.1349999999999998,"5min":9.1349999999999998,"15min":9.1349999999999998},"max":{"1min":23.108000000000001,"5min":170.17400000000001,"15min":170.17400000000001},"last":19.364000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":15.271000000000001,"5min":18.404,"15min":24.379000000000001},"min":{"1min":10.272,"5min":8.9510000000000005,"15min":8.9510000000000005},"max":{"1min":19.027999999999999,"5min":108.045,"15min":108.045},"last":21.654},{"interface":"front","average":{"1min":17.634,"5min":20.315000000000001,"15min":27.440999999999999},"min":{"1min":10.829000000000001,"5min":9.9000000000000004,"15min":9.9000000000000004},"max":{"1min":23.558,"5min":141.97300000000001,"15min":141.97300000000001},"last":16.785}]},{"osd":5,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":16.079000000000001,"5min":20.795999999999999,"15min":26.815999999999999},"min":{"1min":9.9489999999999998,"5min":9.9489999999999998,"15min":9.9489999999999998},"max":{"1min":22.390000000000001,"5min":116.873,"15min":116.873},"last":23.071000000000002},{"interface":"front","average":{"1min":19.495000000000001,"5min":22.710999999999999,"15min":31.870999999999999},"min":{"1min":11.393000000000001,"5min":9.9550000000000001,"15min":9.9550000000000001},"max":{"1min":24.968,"5min":199.773,"15min":199.773},"last":14.414999999999999}]},{"osd":6,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":19.128,"5min":22.716000000000001,"15min":28.771999999999998},"min":{"1min":12.276999999999999,"5min":10.601000000000001,"15min":10.601000000000001},"max":{"1min":23.670999999999999,"5min":131.06299999999999,"15min":131.06299999999999},"last":24.190999999999999},{"interface":"front","average":{"1min":20.283000000000001,"5min":23.718,"15min":31.068000000000001},"min":{"1min":14.736000000000001,"5min":13.375999999999999,"15min":13.375999999999999},"max":{"1min":27.178000000000001,"5min":139.024,"15min":139.024},"last":17.849}]},{"osd":7,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":16.785,"5min":19.227,"15min":26.094999999999999},"min":{"1min":9.577,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":27.773,"5min":127.039,"15min":127.039},"last":14.083},{"interface":"front","average":{"1min":18.727,"5min":21.361999999999998,"15min":27.317},"min":{"1min":11.532,"5min":10.449,"15min":10.449},"max":{"1min":24.533999999999999,"5min":130.88499999999999,"15min":130.88499999999999},"last":15.206}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:53:18.137 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-30T00:47:51.911887+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=47, tm_sec=51, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=53, tm_sec=8, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:53:18.138 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-09-30T00:47:51.911887+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=47, tm_sec=51, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=53, tm_sec=8, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:53:18.138 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-30T00:47:51.911887+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=47, tm_sec=51, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=53, tm_sec=8, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:53:18.138 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-30T00:47:51.911887+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=47, tm_sec=51, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=53, tm_sec=8, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:53:18.138 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-30T00:47:51.911887+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=47, tm_sec=51, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=53, tm_sec=8, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:53:18.138 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-30T00:47:51.911887+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=47, tm_sec=51, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=53, tm_sec=8, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:53:18.139 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-30T00:47:51.911887+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=47, tm_sec=51, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=53, tm_sec=8, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:53:18.139 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-30T00:47:51.911887+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=47, tm_sec=51, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=53, tm_sec=8, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:53:18.139 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T00:53:19.282 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:06:54.993 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:19.287 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:06:54.999 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:19.317 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:06:55.028 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:19.338 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:06:55.025 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:19.342 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:06:55.029 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:19.346 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:06:55.058 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:19.353 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:06:55.065 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:19.367 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:06:55.054 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:19.384 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:06:55.096 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:38.140 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:53:38.284 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.283+0000 7fc635feb700 10 monclient: get_monmap_and_config 2023-09-30T00:53:38.284 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.283+0000 7fc635feb700 10 monclient: build_initial_monmap 2023-09-30T00:53:38.284 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.283+0000 7fc635feb700 10 monclient: monmap: 2023-09-30T00:53:38.285 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:38.285 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:38.285 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:38.284869+0000 2023-09-30T00:53:38.285 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:38.284869+0000 2023-09-30T00:53:38.285 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:38.285 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:38.285 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:38.286 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:38.286 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:38.286 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:38.286 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:38.286 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:38.288 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.284+0000 7fc635feb700 1 Processor -- start 2023-09-30T00:53:38.288 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.284+0000 7fc635feb700 1 -- start start 2023-09-30T00:53:38.288 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.284+0000 7fc635feb700 10 monclient: init 2023-09-30T00:53:38.288 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc635feb700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:38.288 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc635feb700 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-09-30T00:53:38.288 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc635feb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6300b1460 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc635feb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6300b1830 0x7fc6300a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc635feb700 10 monclient(hunting): picked mon.noname-b con 0x7fc6300b1830 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc635feb700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc63013ac20 0x7fc63013b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc635feb700 10 monclient(hunting): picked mon.noname-c con 0x7fc63013ac20 addr v2:172.21.15.135:3300/0 2023-09-30T00:53:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc635feb700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6300a5b20 con 0x7fc6300b1460 2023-09-30T00:53:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc635feb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc635feb700 1 -- --> v2:172.21.15.135:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc630141e40 con 0x7fc63013ac20 2023-09-30T00:53:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc635feb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc635feb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300a64f0 con 0x7fc6300b1830 2023-09-30T00:53:38.289 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc635feb700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc635feb700 10 monclient(hunting): authenticate will time out at 5679.586367s 2023-09-30T00:53:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.285+0000 7fc62effd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6300b1830 0x7fc6300a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.286+0000 7fc62ffff700 1 --2- >> v2:172.21.15.135:3300/0 conn(0x7fc63013ac20 0x7fc63013b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.286+0000 7fc62effd700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6300b1830 0x7fc6300a5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3301/0 says I am v2:172.21.15.40:47928/0 (socket says 172.21.15.40:47928) 2023-09-30T00:53:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.286+0000 7fc62effd700 1 -- 172.21.15.40:0/2718369223 learned_addr learned my addr 172.21.15.40:0/2718369223 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.286+0000 7fc62effd700 10 monclient(hunting): get_auth_request con 0x7fc6300b1830 auth_method 0 2023-09-30T00:53:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.286+0000 7fc62effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.286+0000 7fc62effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.286+0000 7fc62effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.287+0000 7fc62ffff700 10 monclient(hunting): get_auth_request con 0x7fc63013ac20 auth_method 0 2023-09-30T00:53:38.290 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.287+0000 7fc62ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.287+0000 7fc62ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.287+0000 7fc62ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.287+0000 7fc62effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.287+0000 7fc62effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.287+0000 7fc62effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.288+0000 7fc62ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.288+0000 7fc62ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:38.291 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.288+0000 7fc62ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:38.292 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.290+0000 7fc62effd700 10 monclient(hunting): handle_auth_done global_id 4772 payload 274 2023-09-30T00:53:38.292 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.290+0000 7fc62effd700 1 -- 172.21.15.40:0/2718369223 >> v2:172.21.15.135:3300/0 conn(0x7fc63013ac20 msgr2=0x7fc63013b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:38.292 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.290+0000 7fc62effd700 1 --2- 172.21.15.40:0/2718369223 >> v2:172.21.15.135:3300/0 conn(0x7fc63013ac20 0x7fc63013b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.292 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.290+0000 7fc62effd700 1 -- 172.21.15.40:0/2718369223 >> v1:172.21.15.135:6789/0 conn(0x7fc6300b1460 legacy=0x7fc630141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:38.292 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.290+0000 7fc62effd700 10 monclient: _finish_hunting 0 2023-09-30T00:53:38.292 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.290+0000 7fc62effd700 1 monclient: found mon.noname-b 2023-09-30T00:53:38.293 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.290+0000 7fc62effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:38.293 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.290+0000 7fc62effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.40:3301/0 2023-09-30T00:53:38.293 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.290+0000 7fc62effd700 1 -- 172.21.15.40:0/2718369223 --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6300a71f0 con 0x7fc6300b1830 2023-09-30T00:53:38.293 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.292+0000 7fc62effd700 1 --2- 172.21.15.40:0/2718369223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6300b1830 0x7fc6300a5fb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc6180099b0 tx=0x7fc618009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e3ecc00c08c9080 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:38.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc62e7fc700 1 -- 172.21.15.40:0/2718369223 <== mon.2 v2:172.21.15.40:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618009ea0 con 0x7fc6300b1830 2023-09-30T00:53:38.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc62e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:38.298 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc62e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc62e7fc700 10 monclient: dump: 2023-09-30T00:53:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:38.299 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:38.299 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc62e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:53:38.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc62e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:38.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc62e7fc700 1 -- 172.21.15.40:0/2718369223 <== mon.2 v2:172.21.15.40:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61800b820 con 0x7fc6300b1830 2023-09-30T00:53:38.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc635feb700 5 monclient: authenticate success, global_id 4772 2023-09-30T00:53:38.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc62e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:38.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc62e7fc700 1 -- 172.21.15.40:0/2718369223 <== mon.2 v2:172.21.15.40:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61800bbc0 con 0x7fc6300b1830 2023-09-30T00:53:38.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc62e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:38.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc62e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:53:38.300 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc62e7fc700 10 monclient: dump: 2023-09-30T00:53:38.300 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:38.300 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:38.300 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:38.301 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc635feb700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc635feb700 4 set_mon_vals no callback set 2023-09-30T00:53:38.301 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc635feb700 10 monclient: shutdown 2023-09-30T00:53:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc635feb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc635feb700 1 -- 172.21.15.40:0/2718369223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6300b1830 msgr2=0x7fc6300a5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.297+0000 7fc635feb700 1 --2- 172.21.15.40:0/2718369223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6300b1830 0x7fc6300a5fb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc6180099b0 tx=0x7fc618009250 comp rx=0 tx=0).stop 2023-09-30T00:53:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.298+0000 7fc635feb700 1 -- 172.21.15.40:0/2718369223 shutdown_connections 2023-09-30T00:53:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.298+0000 7fc635feb700 1 --2- 172.21.15.40:0/2718369223 >> v2:172.21.15.135:3300/0 conn(0x7fc63013ac20 0x7fc63013b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.298+0000 7fc635feb700 1 --2- 172.21.15.40:0/2718369223 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6300b1830 0x7fc6300a5fb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.298+0000 7fc635feb700 1 -- 172.21.15.40:0/2718369223 >> 172.21.15.40:0/2718369223 conn(0x7fc6300631f0 msgr2=0x7fc6300635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.298+0000 7fc635feb700 1 -- 172.21.15.40:0/2718369223 shutdown_connections 2023-09-30T00:53:38.302 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.298+0000 7fc635feb700 1 -- 172.21.15.40:0/2718369223 wait complete. 2023-09-30T00:53:38.305 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.298+0000 7fc635feb700 10 monclient: build_initial_monmap 2023-09-30T00:53:38.305 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.299+0000 7fc635feb700 10 monclient: monmap: 2023-09-30T00:53:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:38.306 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:38.307 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.299+0000 7fc635feb700 1 librados: starting msgr at 2023-09-30T00:53:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.299+0000 7fc635feb700 1 librados: starting objecter 2023-09-30T00:53:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.299+0000 7fc635feb700 1 Processor -- start 2023-09-30T00:53:38.307 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.299+0000 7fc635feb700 1 -- start start 2023-09-30T00:53:38.308 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.299+0000 7fc635feb700 1 librados: setting wanted keys 2023-09-30T00:53:38.308 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.299+0000 7fc635feb700 1 librados: calling monclient init 2023-09-30T00:53:38.308 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.299+0000 7fc635feb700 10 monclient: init 2023-09-30T00:53:38.308 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:53:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6300b1460 0x7fc6300a9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 10 monclient(hunting): picked mon.noname-c con 0x7fc6300b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6300b1830 0x7fc63013dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 10 monclient(hunting): picked mon.noname-a con 0x7fc6300b1830 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc63013ac20 0x7fc630150be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 10 monclient(hunting): picked mon.noname-b con 0x7fc63013ac20 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300a9c40 con 0x7fc6300b1830 2023-09-30T00:53:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:38.309 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6300aa0d0 con 0x7fc63013ac20 2023-09-30T00:53:38.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:38.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63013e8f0 con 0x7fc6300b1460 2023-09-30T00:53:38.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:38.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.300+0000 7fc635feb700 10 monclient(hunting): authenticate will time out at 5679.601560s 2023-09-30T00:53:38.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62ffff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc63013ac20 0x7fc630150be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:38.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62effd700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6300b1830 0x7fc63013dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:38.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62f7fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6300b1460 0x7fc6300a9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:38.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62ffff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc63013ac20 0x7fc630150be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.135:3300/0 says I am v2:172.21.15.40:59472/0 (socket says 172.21.15.40:59472) 2023-09-30T00:53:38.310 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62ffff700 1 -- 172.21.15.40:0/2793317608 learned_addr learned my addr 172.21.15.40:0/2793317608 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62ffff700 10 monclient(hunting): get_auth_request con 0x7fc63013ac20 auth_method 0 2023-09-30T00:53:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62effd700 10 monclient(hunting): get_auth_request con 0x7fc6300b1830 auth_method 0 2023-09-30T00:53:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62f7fe700 10 monclient(hunting): get_auth_request con 0x7fc6300b1460 auth_method 0 2023-09-30T00:53:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:38.311 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:38.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.301+0000 7fc62f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:38.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.302+0000 7fc62ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:38.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.302+0000 7fc62ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:38.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.302+0000 7fc62ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:38.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.302+0000 7fc62f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:38.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.302+0000 7fc62f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:38.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.302+0000 7fc62f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:38.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.302+0000 7fc62effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:38.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.302+0000 7fc62effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:38.312 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.302+0000 7fc62effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.305+0000 7fc62ffff700 10 monclient(hunting): handle_auth_done global_id 5008 payload 824 2023-09-30T00:53:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.305+0000 7fc62ffff700 1 -- 172.21.15.40:0/2793317608 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6300b1460 msgr2=0x7fc6300a9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.305+0000 7fc62ffff700 1 --2- 172.21.15.40:0/2793317608 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6300b1460 0x7fc6300a9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.305+0000 7fc62ffff700 1 -- 172.21.15.40:0/2793317608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6300b1830 msgr2=0x7fc63013dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.305+0000 7fc62ffff700 1 --2- 172.21.15.40:0/2793317608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6300b1830 0x7fc63013dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.305+0000 7fc62ffff700 10 monclient: _finish_hunting 0 2023-09-30T00:53:38.315 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.305+0000 7fc62ffff700 1 monclient: found mon.noname-b 2023-09-30T00:53:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.305+0000 7fc62ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.305+0000 7fc62ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.305+0000 7fc62ffff700 1 -- 172.21.15.40:0/2793317608 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6180094f0 con 0x7fc63013ac20 2023-09-30T00:53:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.308+0000 7fc62ffff700 1 --2- 172.21.15.40:0/2793317608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc63013ac20 0x7fc630150be0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc628009df0 tx=0x7fc62800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.308+0000 7fc6277fe700 10 client.?.objecter ms_handle_connect 0x7fc63013ac20 2023-09-30T00:53:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.308+0000 7fc6277fe700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc6277fe700 1 -- 172.21.15.40:0/2793317608 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62800caa0 con 0x7fc63013ac20 2023-09-30T00:53:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc6277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:38.316 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc6277fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc6277fe700 10 monclient: dump: 2023-09-30T00:53:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:38.317 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc6277fe700 10 monclient: _finish_auth 0 2023-09-30T00:53:38.317 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc6277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:38.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc6277fe700 1 -- 172.21.15.40:0/2793317608 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62800f040 con 0x7fc63013ac20 2023-09-30T00:53:38.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc635feb700 5 monclient: authenticate success, global_id 5008 2023-09-30T00:53:38.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc6277fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:38.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc6277fe700 1 -- 172.21.15.40:0/2793317608 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62800ce70 con 0x7fc63013ac20 2023-09-30T00:53:38.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc635feb700 10 monclient: _renew_subs 2023-09-30T00:53:38.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc635feb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:38.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc635feb700 1 -- 172.21.15.40:0/2793317608 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6300a8c60 con 0x7fc63013ac20 2023-09-30T00:53:38.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc6277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:38.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc6277fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:38.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc62dffb700 4 set_mon_vals no callback set 2023-09-30T00:53:38.318 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc6277fe700 10 monclient: dump: 2023-09-30T00:53:38.319 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:38.319 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:38.319 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.319 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.319 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:38.319 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:38.319 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:38.319 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:38.319 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:38.319 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:38.319 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc635feb700 10 client.5008.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:38.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc635feb700 10 monclient: _renew_subs 2023-09-30T00:53:38.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc635feb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:38.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc635feb700 1 -- 172.21.15.40:0/2793317608 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc630151360 con 0x7fc63013ac20 2023-09-30T00:53:38.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.313+0000 7fc635feb700 1 librados: init done 2023-09-30T00:53:38.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.314+0000 7fc635feb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:38.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.314+0000 7fc635feb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:38.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.314+0000 7fc635feb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:38.320 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.314+0000 7fc635feb700 1 -- 172.21.15.40:0/2793317608 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6300a9df0 con 0x7fc63013ac20 2023-09-30T00:53:38.563 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.563+0000 7fc6277fe700 1 -- 172.21.15.40:0/2793317608 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fc628014690 con 0x7fc63013ac20 2023-09-30T00:53:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.563+0000 7fc6277fe700 1 --2- 172.21.15.40:0/2793317608 >> v2:172.21.15.135:6800/116426 conn(0x7fc610068890 0x7fc61006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.564+0000 7fc62f7fe700 1 --2- 172.21.15.40:0/2793317608 >> v2:172.21.15.135:6800/116426 conn(0x7fc610068890 0x7fc61006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:38.564 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.564+0000 7fc62f7fe700 10 monclient: get_auth_request con 0x7fc610068890 auth_method 0 2023-09-30T00:53:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.564+0000 7fc62f7fe700 1 --2- 172.21.15.40:0/2793317608 >> v2:172.21.15.135:6800/116426 conn(0x7fc610068890 0x7fc61006ad40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc6200039a0 tx=0x7fc620007ea0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:38.565 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.564+0000 7fc6277fe700 10 client.5008.objecter ms_handle_connect 0x7fc610068890 2023-09-30T00:53:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.567+0000 7fc6277fe700 1 -- 172.21.15.40:0/2793317608 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fc6280569b0 con 0x7fc63013ac20 2023-09-30T00:53:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.567+0000 7fc6277fe700 10 client.5008.objecter ms_dispatch 0x7fc630001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.567+0000 7fc6277fe700 3 client.5008.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:38.568 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.567+0000 7fc6277fe700 3 client.5008.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:38.569 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.567+0000 7fc6277fe700 20 client.5008.objecter dump_active .. 0 homeless 2023-09-30T00:53:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.577+0000 7fc6277fe700 1 -- 172.21.15.40:0/2793317608 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc628052420 con 0x7fc63013ac20 2023-09-30T00:53:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.577+0000 7fc6277fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:38.577 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.577+0000 7fc6277fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:38.784 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.783+0000 7fc635feb700 1 -- 172.21.15.40:0/2793317608 --> v2:172.21.15.135:6800/116426 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc6300a9df0 con 0x7fc610068890 2023-09-30T00:53:38.787 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.787+0000 7fc6277fe700 1 -- 172.21.15.40:0/2793317608 <== mgr.4099 v2:172.21.15.135:6800/116426 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70516 (secure 0 0 0) 0x7fc6300a9df0 con 0x7fc610068890 2023-09-30T00:53:38.788 INFO:teuthology.orchestra.run.smithi040.stdout: 2023-09-30T00:53:38.790 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 10 librados: watch_flush enter 2023-09-30T00:53:38.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 10 librados: watch_flush exit 2023-09-30T00:53:38.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 20 client.5008.objecter shutdown clearing up homeless session... 2023-09-30T00:53:38.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 10 client.5008.objecter successfully canceled tick 2023-09-30T00:53:38.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 1 -- 172.21.15.40:0/2793317608 >> v2:172.21.15.135:6800/116426 conn(0x7fc610068890 msgr2=0x7fc61006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:38.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 1 --2- 172.21.15.40:0/2793317608 >> v2:172.21.15.135:6800/116426 conn(0x7fc610068890 0x7fc61006ad40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc6200039a0 tx=0x7fc620007ea0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 10 monclient: shutdown 2023-09-30T00:53:38.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:38.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 1 -- 172.21.15.40:0/2793317608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc63013ac20 msgr2=0x7fc630150be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:38.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 1 --2- 172.21.15.40:0/2793317608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc63013ac20 0x7fc630150be0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc628009df0 tx=0x7fc62800b040 comp rx=0 tx=0).stop 2023-09-30T00:53:38.791 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 1 -- 172.21.15.40:0/2793317608 shutdown_connections 2023-09-30T00:53:38.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 1 --2- 172.21.15.40:0/2793317608 >> v2:172.21.15.135:6800/116426 conn(0x7fc610068890 0x7fc61006ad40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 1 --2- 172.21.15.40:0/2793317608 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7fc63013ac20 0x7fc630150be0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 1 --2- 172.21.15.40:0/2793317608 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7fc6300b1460 0x7fc6300a9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 1 --2- 172.21.15.40:0/2793317608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6300b1830 0x7fc63013dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 1 -- 172.21.15.40:0/2793317608 >> 172.21.15.40:0/2793317608 conn(0x7fc6300631f0 msgr2=0x7fc630144b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:38.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 1 -- 172.21.15.40:0/2793317608 shutdown_connections 2023-09-30T00:53:38.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.789+0000 7fc635feb700 1 -- 172.21.15.40:0/2793317608 wait complete. 2023-09-30T00:53:38.792 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.790+0000 7fc635feb700 1 librados: shutdown 2023-09-30T00:53:38.793 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2023-09-30T00:53:38.803 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2023-09-30T00:53:37.945083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43216,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40704,"kb_avail":754931504,"statfs":{"total":773094113280,"available":773049860096,"internally_reserved":0,"allocated":2506752,"data_stored":1383352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41680896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002578"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:53:30.884757+0000","last_change":"2023-09-30T00:53:30.883872+0000","last_active":"2023-09-30T00:53:30.884757+0000","last_peered":"2023-09-30T00:53:30.884757+0000","last_clean":"2023-09-30T00:53:30.884757+0000","last_became_active":"2023-09-30T00:47:54.672127+0000","last_became_peered":"2023-09-30T00:47:54.672127+0000","last_unstale":"2023-09-30T00:53:30.884757+0000","last_undegraded":"2023-09-30T00:53:30.884757+0000","last_fullsized":"2023-09-30T00:53:30.884757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:53:30.881171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:53:30.881171+0000","last_clean_scrub_stamp":"2023-09-30T00:53:30.881171+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:14:39.534045+0000","scrub_duration":0.044837799999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:53:22.648265+0000","last_change":"2023-09-30T00:53:22.635654+0000","last_active":"2023-09-30T00:53:22.648265+0000","last_peered":"2023-09-30T00:53:22.648265+0000","last_clean":"2023-09-30T00:53:22.648265+0000","last_became_active":"2023-09-30T00:47:54.702773+0000","last_became_peered":"2023-09-30T00:47:54.702773+0000","last_unstale":"2023-09-30T00:53:22.648265+0000","last_undegraded":"2023-09-30T00:53:22.648265+0000","last_fullsized":"2023-09-30T00:53:22.648265+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:53:22.625552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:53:22.625552+0000","last_clean_scrub_stamp":"2023-09-30T00:53:22.625552+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:12:30.829656+0000","scrub_duration":0.48125541999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:53:26.978004+0000","last_change":"2023-09-30T00:53:26.977181+0000","last_active":"2023-09-30T00:53:26.978004+0000","last_peered":"2023-09-30T00:53:26.978004+0000","last_clean":"2023-09-30T00:53:26.978004+0000","last_became_active":"2023-09-30T00:47:54.139464+0000","last_became_peered":"2023-09-30T00:47:54.139464+0000","last_unstale":"2023-09-30T00:53:26.978004+0000","last_undegraded":"2023-09-30T00:53:26.978004+0000","last_fullsized":"2023-09-30T00:53:26.978004+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:53:26.974388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:53:26.974388+0000","last_clean_scrub_stamp":"2023-09-30T00:53:26.974388+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:52:49.110533+0000","scrub_duration":0.090487052999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:53:22.300776+0000","last_change":"2023-09-30T00:53:22.292282+0000","last_active":"2023-09-30T00:53:22.300776+0000","last_peered":"2023-09-30T00:53:22.300776+0000","last_clean":"2023-09-30T00:53:22.300776+0000","last_became_active":"2023-09-30T00:47:53.951882+0000","last_became_peered":"2023-09-30T00:47:53.951882+0000","last_unstale":"2023-09-30T00:53:22.300776+0000","last_undegraded":"2023-09-30T00:53:22.300776+0000","last_fullsized":"2023-09-30T00:53:22.300776+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:53:22.279388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:53:22.279388+0000","last_clean_scrub_stamp":"2023-09-30T00:53:22.279388+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:19:47.006297+0000","scrub_duration":0.33666995599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'79","reported_seq":127,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:53:13.271839+0000","last_change":"2023-09-30T00:53:13.262553+0000","last_active":"2023-09-30T00:53:13.271839+0000","last_peered":"2023-09-30T00:53:13.271839+0000","last_clean":"2023-09-30T00:53:13.271839+0000","last_became_active":"2023-09-30T00:47:46.406306+0000","last_became_peered":"2023-09-30T00:47:46.406306+0000","last_unstale":"2023-09-30T00:53:13.271839+0000","last_undegraded":"2023-09-30T00:53:13.271839+0000","last_fullsized":"2023-09-30T00:53:13.271839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'79","last_scrub_stamp":"2023-09-30T00:53:13.250386+0000","last_deep_scrub":"25'79","last_deep_scrub_stamp":"2023-09-30T00:53:13.250386+0000","last_clean_scrub_stamp":"2023-09-30T00:53:13.250386+0000","objects_scrubbed":2,"log_size":79,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:21:58.427216+0000","scrub_duration":0.73232861699999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:53:25.065318+0000","last_change":"2023-09-30T00:53:25.064540+0000","last_active":"2023-09-30T00:53:25.065318+0000","last_peered":"2023-09-30T00:53:25.065318+0000","last_clean":"2023-09-30T00:53:25.065318+0000","last_became_active":"2023-09-30T00:47:54.710752+0000","last_became_peered":"2023-09-30T00:47:54.710752+0000","last_unstale":"2023-09-30T00:53:25.065318+0000","last_undegraded":"2023-09-30T00:53:25.065318+0000","last_fullsized":"2023-09-30T00:53:25.065318+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:53:25.057203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:53:25.057203+0000","last_clean_scrub_stamp":"2023-09-30T00:53:25.057203+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:37:47.637208+0000","scrub_duration":0.186038288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:53:18.275226+0000","last_change":"2023-09-30T00:53:18.265458+0000","last_active":"2023-09-30T00:53:18.275226+0000","last_peered":"2023-09-30T00:53:18.275226+0000","last_clean":"2023-09-30T00:53:18.275226+0000","last_became_active":"2023-09-30T00:47:53.761460+0000","last_became_peered":"2023-09-30T00:47:53.761460+0000","last_unstale":"2023-09-30T00:53:18.275226+0000","last_undegraded":"2023-09-30T00:53:18.275226+0000","last_fullsized":"2023-09-30T00:53:18.275226+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:53:18.258541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:53:18.258541+0000","last_clean_scrub_stamp":"2023-09-30T00:53:18.258541+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:45:11.750457+0000","scrub_duration":0.23803902099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:53:23.939591+0000","last_change":"2023-09-30T00:53:23.938807+0000","last_active":"2023-09-30T00:53:23.939591+0000","last_peered":"2023-09-30T00:53:23.939591+0000","last_clean":"2023-09-30T00:53:23.939591+0000","last_became_active":"2023-09-30T00:47:54.279706+0000","last_became_peered":"2023-09-30T00:47:54.279706+0000","last_unstale":"2023-09-30T00:53:23.939591+0000","last_undegraded":"2023-09-30T00:53:23.939591+0000","last_fullsized":"2023-09-30T00:53:23.939591+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:53:23.936120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:53:23.936120+0000","last_clean_scrub_stamp":"2023-09-30T00:53:23.936120+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T11:03:49.053996+0000","scrub_duration":0.060217390000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T00:53:16.255173+0000","last_change":"2023-09-30T00:53:16.242192+0000","last_active":"2023-09-30T00:53:16.255173+0000","last_peered":"2023-09-30T00:53:16.255173+0000","last_clean":"2023-09-30T00:53:16.255173+0000","last_became_active":"2023-09-30T00:47:54.305507+0000","last_became_peered":"2023-09-30T00:47:54.305507+0000","last_unstale":"2023-09-30T00:53:16.255173+0000","last_undegraded":"2023-09-30T00:53:16.255173+0000","last_fullsized":"2023-09-30T00:53:16.255173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:53:16.235305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:53:16.235305+0000","last_clean_scrub_stamp":"2023-09-30T00:53:16.235305+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:29:02.885823+0000","scrub_duration":0.37016202599999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":79,"num_write":114,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":20,"seq":85899345995,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":13.013999999999999,"5min":12.769,"15min":13.343},"min":{"1min":7.7699999999999996,"5min":6.6319999999999997,"15min":6.6319999999999997},"max":{"1min":20.431999999999999,"5min":26.166,"15min":26.166},"last":10.053000000000001},{"interface":"front","average":{"1min":10.711,"5min":11.778,"15min":13.131},"min":{"1min":6.8440000000000003,"5min":6.8440000000000003,"15min":6.8440000000000003},"max":{"1min":15.452,"5min":32.808,"15min":32.808},"last":12.718999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":12.897,"5min":13.236000000000001,"15min":14.891999999999999},"min":{"1min":9.0199999999999996,"5min":7.8250000000000002,"15min":7.8250000000000002},"max":{"1min":21.581,"5min":31.925999999999998,"15min":31.925999999999998},"last":10.823},{"interface":"front","average":{"1min":15.115,"5min":16.175999999999998,"15min":17.780000000000001},"min":{"1min":9.3369999999999997,"5min":8.4629999999999992,"15min":8.4629999999999992},"max":{"1min":23.300000000000001,"5min":42.737000000000002,"15min":42.737000000000002},"last":14.725}]},{"osd":2,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":14.624000000000001,"5min":16.120999999999999,"15min":17.459},"min":{"1min":8.9489999999999998,"5min":8.3469999999999995,"15min":8.3469999999999995},"max":{"1min":22.853000000000002,"5min":36.436999999999998,"15min":36.436999999999998},"last":12.047000000000001},{"interface":"front","average":{"1min":18.524999999999999,"5min":19.745999999999999,"15min":25.524999999999999},"min":{"1min":11.426,"5min":10.092000000000001,"15min":10.092000000000001},"max":{"1min":22.728999999999999,"5min":164.05600000000001,"15min":164.05600000000001},"last":17.178000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":15.416,"5min":17.084,"15min":17.538},"min":{"1min":10.166,"5min":8.9570000000000007,"15min":8.9570000000000007},"max":{"1min":21.091999999999999,"5min":36.740000000000002,"15min":36.740000000000002},"last":14.233000000000001},{"interface":"front","average":{"1min":13.753,"5min":15.188000000000001,"15min":17.27},"min":{"1min":8.1069999999999993,"5min":7.4329999999999998,"15min":7.4329999999999998},"max":{"1min":18.597999999999999,"5min":39.865000000000002,"15min":39.865000000000002},"last":20.309999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":18.372,"5min":19.262,"15min":21.207000000000001},"min":{"1min":12.487,"5min":10.294,"15min":10.294},"max":{"1min":22.335999999999999,"5min":47.259,"15min":47.259},"last":15.606999999999999},{"interface":"front","average":{"1min":18.518000000000001,"5min":19.602,"15min":21.158000000000001},"min":{"1min":10.493,"5min":9.7059999999999995,"15min":9.7059999999999995},"max":{"1min":23.382999999999999,"5min":46.417999999999999,"15min":46.417999999999999},"last":18.77}]},{"osd":6,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":20.271000000000001,"5min":20.856999999999999,"15min":22.073},"min":{"1min":13.49,"5min":11.042999999999999,"15min":11.042999999999999},"max":{"1min":25.059999999999999,"5min":49.011000000000003,"15min":49.011000000000003},"last":21.068999999999999},{"interface":"front","average":{"1min":16.920999999999999,"5min":17.010999999999999,"15min":18.483000000000001},"min":{"1min":8.9130000000000003,"5min":8.9130000000000003,"15min":8.9130000000000003},"max":{"1min":24.98,"5min":44.369,"15min":44.369},"last":22.687999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":19.391999999999999,"15min":21.218},"min":{"1min":13.249000000000001,"5min":12.590999999999999,"15min":12.590999999999999},"max":{"1min":25.725000000000001,"5min":39.171999999999997,"15min":39.171999999999997},"last":21.765000000000001},{"interface":"front","average":{"1min":19.501999999999999,"5min":21.446999999999999,"15min":23.422000000000001},"min":{"1min":13.321999999999999,"5min":11.161,"15min":11.161},"max":{"1min":24.439,"5min":41.865000000000002,"15min":41.865000000000002},"last":17.960000000000001}]}]},{"osd":2,"up_from":20,"seq":85899345994,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":12.666,"5min":12.167,"15min":12.167},"min":{"1min":6.9450000000000003,"5min":6.6920000000000002,"15min":6.6920000000000002},"max":{"1min":20.530999999999999,"5min":22.748999999999999,"15min":22.748999999999999},"last":9.9139999999999997},{"interface":"front","average":{"1min":12.608000000000001,"5min":13.143000000000001,"15min":14.31},"min":{"1min":9.4100000000000001,"5min":6.319,"15min":6.319},"max":{"1min":21.957000000000001,"5min":31.885000000000002,"15min":31.885000000000002},"last":15.993}]},{"osd":1,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":15.48,"5min":17.213000000000001,"15min":22.105},"min":{"1min":9.6579999999999995,"5min":8.718,"15min":8.718},"max":{"1min":21.058,"5min":125.67700000000001,"15min":125.67700000000001},"last":7.8739999999999997},{"interface":"front","average":{"1min":16.795999999999999,"5min":17.684999999999999,"15min":24.204999999999998},"min":{"1min":10.210000000000001,"5min":8.0370000000000008,"15min":8.0370000000000008},"max":{"1min":24.420999999999999,"5min":156.821,"15min":156.821},"last":15.228}]},{"osd":3,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":10.301,"5min":18.065999999999999,"15min":34.295999999999999},"min":{"1min":7.6210000000000004,"5min":7.6210000000000004,"15min":7.6210000000000004},"max":{"1min":14.249000000000001,"5min":513.32600000000002,"15min":513.32600000000002},"last":21.859999999999999},{"interface":"front","average":{"1min":17.056999999999999,"5min":24.167999999999999,"15min":42.537999999999997},"min":{"1min":11.009,"5min":10.308,"15min":10.308},"max":{"1min":23.140999999999998,"5min":598.48900000000003,"15min":598.48900000000003},"last":17.686}]},{"osd":4,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":14.541,"5min":16.052,"15min":17.638999999999999},"min":{"1min":9.1530000000000005,"5min":8.8859999999999992,"15min":8.8859999999999992},"max":{"1min":24.239999999999998,"5min":31.702000000000002,"15min":31.702000000000002},"last":8.9160000000000004},{"interface":"front","average":{"1min":16.594000000000001,"5min":16.542000000000002,"15min":17.376999999999999},"min":{"1min":8.6750000000000007,"5min":8.6750000000000007,"15min":8.6750000000000007},"max":{"1min":23.922999999999998,"5min":37.073999999999998,"15min":37.073999999999998},"last":17.266999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":19.826000000000001,"5min":20.279,"15min":21.254000000000001},"min":{"1min":13.452999999999999,"5min":9.1440000000000001,"15min":9.1440000000000001},"max":{"1min":25.805,"5min":38.981000000000002,"15min":38.981000000000002},"last":12.318},{"interface":"front","average":{"1min":18.370999999999999,"5min":20.248000000000001,"15min":23.968},"min":{"1min":13.335000000000001,"5min":9.8949999999999996,"15min":9.8949999999999996},"max":{"1min":24.638999999999999,"5min":74.510000000000005,"15min":74.510000000000005},"last":19.622}]},{"osd":6,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":18.266999999999999,"5min":17.795000000000002,"15min":19.167000000000002},"min":{"1min":13.561,"5min":7.6150000000000002,"15min":7.6150000000000002},"max":{"1min":24.135999999999999,"5min":40.875999999999998,"15min":40.875999999999998},"last":18.899999999999999},{"interface":"front","average":{"1min":22.277000000000001,"5min":21.561,"15min":23.045999999999999},"min":{"1min":15.621,"5min":9.5809999999999995,"15min":9.5809999999999995},"max":{"1min":26.98,"5min":42.539000000000001,"15min":42.539000000000001},"last":19.904}]},{"osd":7,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":20.937000000000001,"5min":21.863,"15min":24.445},"min":{"1min":14.688000000000001,"5min":12.295999999999999,"15min":12.295999999999999},"max":{"1min":26.420999999999999,"5min":76.484999999999999,"15min":76.484999999999999},"last":22.673999999999999},{"interface":"front","average":{"1min":19.401,"5min":20.391999999999999,"15min":22.324000000000002},"min":{"1min":14.641999999999999,"5min":10.223000000000001,"15min":10.223000000000001},"max":{"1min":28.007999999999999,"5min":55.265999999999998,"15min":55.265999999999998},"last":11.417}]}]},{"osd":5,"up_from":15,"seq":64424509515,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5920,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365920,"statfs":{"total":96636764160,"available":96630702080,"internally_reserved":0,"allocated":614400,"data_stored":471299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":10.923,"5min":10.705,"15min":11.093},"min":{"1min":7.1210000000000004,"5min":5.6130000000000004,"15min":5.6130000000000004},"max":{"1min":18.591999999999999,"5min":23.507000000000001,"15min":23.507000000000001},"last":14.779999999999999},{"interface":"front","average":{"1min":11.551,"5min":10.593999999999999,"15min":10.163},"min":{"1min":7.1470000000000002,"5min":5.2130000000000001,"15min":5.2130000000000001},"max":{"1min":20.713000000000001,"5min":27.425000000000001,"15min":27.425000000000001},"last":10.132999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.073,"5min":12.210000000000001,"15min":12.234},"min":{"1min":6.8369999999999997,"5min":6.6619999999999999,"15min":6.6619999999999999},"max":{"1min":15.596,"5min":31.512,"15min":31.512},"last":12.003},{"interface":"front","average":{"1min":16.539000000000001,"5min":19.786999999999999,"15min":26.213000000000001},"min":{"1min":11.753,"5min":10.006,"15min":10.006},"max":{"1min":21.158000000000001,"5min":232.38800000000001,"15min":232.38800000000001},"last":19.981999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":9.3290000000000006,"5min":10.186,"15min":9.9290000000000003},"min":{"1min":7.0999999999999996,"5min":6.9909999999999997,"15min":6.9909999999999997},"max":{"1min":15.125999999999999,"5min":22.271000000000001,"15min":22.271000000000001},"last":8.3320000000000007},{"interface":"front","average":{"1min":16.318999999999999,"5min":16.466000000000001,"15min":17.452999999999999},"min":{"1min":12.247999999999999,"5min":8.3810000000000002,"15min":8.3810000000000002},"max":{"1min":19.457000000000001,"5min":30.27,"15min":30.27},"last":16.66}]},{"osd":3,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.276,"5min":13.022,"15min":14.42},"min":{"1min":7.1829999999999998,"5min":7.1829999999999998,"15min":7.1829999999999998},"max":{"1min":17.137,"5min":42.023000000000003,"15min":42.023000000000003},"last":7.8719999999999999},{"interface":"front","average":{"1min":14.127000000000001,"5min":14.695,"15min":16.593},"min":{"1min":9.5350000000000001,"5min":7.1029999999999998,"15min":7.1029999999999998},"max":{"1min":21.387,"5min":46.786000000000001,"15min":46.786000000000001},"last":14.1}]},{"osd":4,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":15.722,"5min":14.898,"15min":14.827999999999999},"min":{"1min":10.923,"5min":6.7380000000000004,"15min":6.7380000000000004},"max":{"1min":21.215,"5min":23.788,"15min":23.788},"last":16.026},{"interface":"front","average":{"1min":17.538,"5min":18.876000000000001,"15min":22.504000000000001},"min":{"1min":11.625,"5min":9.2300000000000004,"15min":9.2300000000000004},"max":{"1min":20.460000000000001,"5min":112.548,"15min":112.548},"last":18.584}]},{"osd":6,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":13.363,"5min":14.715999999999999,"15min":17.433},"min":{"1min":6.819,"5min":6.819,"15min":6.819},"max":{"1min":22.664000000000001,"5min":106.333,"15min":106.333},"last":12.606},{"interface":"front","average":{"1min":16.849,"5min":16.956,"15min":19.303000000000001},"min":{"1min":9.9529999999999994,"5min":8.2769999999999992,"15min":8.2769999999999992},"max":{"1min":23.702999999999999,"5min":103.483,"15min":103.483},"last":12.342000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:52:48 2023","interfaces":[{"interface":"back","average":{"1min":17.23,"5min":21.253,"15min":29.43},"min":{"1min":11.779999999999999,"5min":10.369999999999999,"15min":10.369999999999999},"max":{"1min":22.402999999999999,"5min":253.74799999999999,"15min":253.74799999999999},"last":17.959},{"interface":"front","average":{"1min":17.359999999999999,"5min":20.033999999999999,"15min":30.213999999999999},"min":{"1min":9.1869999999999994,"5min":9.0310000000000006,"15min":9.0310000000000006},"max":{"1min":21.940000000000001,"5min":328.31799999999998,"15min":328.31799999999998},"last":21.052}]}]},{"osd":4,"up_from":15,"seq":64424509514,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":9.6129999999999995,"5min":10.558,"15min":12.276},"min":{"1min":6.4080000000000004,"5min":4.556,"15min":4.556},"max":{"1min":19.038,"5min":78.191000000000003,"15min":78.191000000000003},"last":8.6859999999999999},{"interface":"front","average":{"1min":11.808,"5min":11.657,"15min":13.117000000000001},"min":{"1min":6.4189999999999996,"5min":5.5439999999999996,"15min":5.5439999999999996},"max":{"1min":21.591999999999999,"5min":73.787000000000006,"15min":73.787000000000006},"last":7.6849999999999996}]},{"osd":1,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":11.846,"5min":13.288,"15min":15.319000000000001},"min":{"1min":6.6219999999999999,"5min":6.6219999999999999,"15min":6.6219999999999999},"max":{"1min":19.167999999999999,"5min":79.558000000000007,"15min":79.558000000000007},"last":15.236000000000001},{"interface":"front","average":{"1min":13.247999999999999,"5min":14.374000000000001,"15min":16.331},"min":{"1min":7.1130000000000004,"5min":7.1130000000000004,"15min":7.1130000000000004},"max":{"1min":18.385999999999999,"5min":82.048000000000002,"15min":82.048000000000002},"last":13.218999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":15.504,"5min":16.736999999999998,"15min":18.332000000000001},"min":{"1min":9.4640000000000004,"5min":8.2889999999999997,"15min":8.2889999999999997},"max":{"1min":20.145,"5min":98.372,"15min":98.372},"last":16.869},{"interface":"front","average":{"1min":14.375999999999999,"5min":15.202999999999999,"15min":16.692},"min":{"1min":8.0009999999999994,"5min":7.2750000000000004,"15min":7.2750000000000004},"max":{"1min":22.282,"5min":96.753,"15min":96.753},"last":14.441000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":16.495999999999999,"5min":17.460000000000001,"15min":19.975999999999999},"min":{"1min":9.0860000000000003,"5min":8.3629999999999995,"15min":8.3629999999999995},"max":{"1min":20.835000000000001,"5min":100.212,"15min":100.212},"last":19.917000000000002},{"interface":"front","average":{"1min":11.829000000000001,"5min":12.417,"15min":14.603},"min":{"1min":6.8029999999999999,"5min":6.7729999999999997,"15min":6.7729999999999997},"max":{"1min":19.068999999999999,"5min":69.975999999999999,"15min":69.975999999999999},"last":8.4030000000000005}]},{"osd":5,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":16.097000000000001,"5min":14.657999999999999,"15min":18.02},"min":{"1min":7.9880000000000004,"5min":7.3070000000000004,"15min":7.3070000000000004},"max":{"1min":20.745999999999999,"5min":129.88,"15min":129.88},"last":19.013000000000002},{"interface":"front","average":{"1min":15.327999999999999,"5min":18.414000000000001,"15min":21.858000000000001},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":22.792999999999999,"5min":101.876,"15min":101.876},"last":9.9350000000000005}]},{"osd":6,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":19.172000000000001,"5min":21.908999999999999,"15min":28.599},"min":{"1min":14.031000000000001,"5min":11.853999999999999,"15min":11.853999999999999},"max":{"1min":23.716999999999999,"5min":199.161,"15min":199.161},"last":20.803000000000001},{"interface":"front","average":{"1min":13.936999999999999,"5min":17.481000000000002,"15min":24.242999999999999},"min":{"1min":9.1899999999999995,"5min":8.5060000000000002,"15min":8.5060000000000002},"max":{"1min":23.106000000000002,"5min":205.011,"15min":205.011},"last":10.842000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":16.001000000000001,"5min":18.004999999999999,"15min":22.398},"min":{"1min":9.9570000000000007,"5min":8.2110000000000003,"15min":8.2110000000000003},"max":{"1min":22.222000000000001,"5min":140.71000000000001,"15min":140.71000000000001},"last":13.65},{"interface":"front","average":{"1min":16.390999999999998,"5min":17.071999999999999,"15min":19.221},"min":{"1min":10.544,"5min":5.6509999999999998,"15min":5.6509999999999998},"max":{"1min":23.492999999999999,"5min":101.80500000000001,"15min":101.80500000000001},"last":17.699000000000002}]}]},{"osd":6,"up_from":14,"seq":60129542219,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.535,"5min":13.951000000000001,"15min":22.998000000000001},"min":{"1min":6.5380000000000003,"5min":4.5990000000000002,"15min":4.5990000000000002},"max":{"1min":17.355,"5min":350.05700000000002,"15min":350.05700000000002},"last":8.3420000000000005},{"interface":"front","average":{"1min":9.7520000000000007,"5min":13.462999999999999,"15min":23.047999999999998},"min":{"1min":5.383,"5min":5.383,"15min":5.383},"max":{"1min":18.678000000000001,"5min":352.40899999999999,"15min":352.40899999999999},"last":12.824}]},{"osd":1,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":12.308,"5min":18.155999999999999,"15min":33.904000000000003},"min":{"1min":6.367,"5min":5.5700000000000003,"15min":5.5700000000000003},"max":{"1min":19.513000000000002,"5min":332.67399999999998,"15min":332.67399999999998},"last":11.477},{"interface":"front","average":{"1min":12.640000000000001,"5min":19.193999999999999,"15min":37.317999999999998},"min":{"1min":7.8559999999999999,"5min":7.2629999999999999,"15min":7.2629999999999999},"max":{"1min":18.895,"5min":359.57900000000001,"15min":359.57900000000001},"last":10.603999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":11.337999999999999,"5min":18.093,"15min":29.870000000000001},"min":{"1min":6.907,"5min":6.593,"15min":6.593},"max":{"1min":22.527999999999999,"5min":420.69299999999998,"15min":420.69299999999998},"last":9.7189999999999994},{"interface":"front","average":{"1min":16.591999999999999,"5min":17.097000000000001,"15min":26.565000000000001},"min":{"1min":10.023,"5min":6.0990000000000002,"15min":6.0990000000000002},"max":{"1min":19.998000000000001,"5min":421.416,"15min":421.416},"last":20.562000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:53:01 2023","interfaces":[{"interface":"back","average":{"1min":13.147,"5min":14.747,"15min":17.933},"min":{"1min":7.4450000000000003,"5min":5.9710000000000001,"15min":5.9710000000000001},"max":{"1min":21.074999999999999,"5min":142.905,"15min":142.905},"last":15.612},{"interface":"front","average":{"1min":17.030999999999999,"5min":19.399999999999999,"15min":23.738},"min":{"1min":13.157,"5min":11.380000000000001,"15min":11.380000000000001},"max":{"1min":19.614000000000001,"5min":177.86199999999999,"15min":177.86199999999999},"last":14.691000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":14.191000000000001,"5min":15.872999999999999,"15min":19.667999999999999},"min":{"1min":9.0670000000000002,"5min":7.0270000000000001,"15min":7.0270000000000001},"max":{"1min":20.523,"5min":164.02699999999999,"15min":164.02699999999999},"last":13.737},{"interface":"front","average":{"1min":17.466999999999999,"5min":19.172999999999998,"15min":23.913},"min":{"1min":10.384,"5min":6.7400000000000002,"15min":6.7400000000000002},"max":{"1min":22.350000000000001,"5min":183.18000000000001,"15min":183.18000000000001},"last":18.623000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":12.683999999999999,"5min":15.390000000000001,"15min":19.122},"min":{"1min":7.5229999999999997,"5min":7.101,"15min":7.101},"max":{"1min":17.652000000000001,"5min":163.00700000000001,"15min":163.00700000000001},"last":19.279},{"interface":"front","average":{"1min":15.941000000000001,"5min":17.882000000000001,"15min":22.271999999999998},"min":{"1min":11.317,"5min":7.2770000000000001,"15min":7.2770000000000001},"max":{"1min":21.721,"5min":177.17400000000001,"15min":177.17400000000001},"last":17.635999999999999}]},{"osd":7,"last update":"Sat Sep 30 00:52:50 2023","interfaces":[{"interface":"back","average":{"1min":18.608000000000001,"5min":20.460000000000001,"15min":24.350000000000001},"min":{"1min":12.318,"5min":8.8469999999999995,"15min":8.8469999999999995},"max":{"1min":21.919,"5min":179.86500000000001,"15min":179.86500000000001},"last":16.588999999999999},{"interface":"front","average":{"1min":14.218,"5min":17.603999999999999,"15min":21.802},"min":{"1min":7.4619999999999997,"5min":7.4619999999999997,"15min":7.4619999999999997},"max":{"1min":19.574000000000002,"5min":164.58099999999999,"15min":164.58099999999999},"last":21.34}]}]},{"osd":7,"up_from":15,"seq":64424509514,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":9.2799999999999994,"5min":10.220000000000001,"15min":11.925000000000001},"min":{"1min":4.2999999999999998,"5min":4.2999999999999998,"15min":4.2999999999999998},"max":{"1min":17.898,"5min":52.545999999999999,"15min":52.545999999999999},"last":9.6489999999999991},{"interface":"front","average":{"1min":10.803000000000001,"5min":11.276999999999999,"15min":13.035},"min":{"1min":7.5449999999999999,"5min":6.6180000000000003,"15min":6.6180000000000003},"max":{"1min":17.611000000000001,"5min":60.087000000000003,"15min":60.087000000000003},"last":10.638}]},{"osd":1,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":12.558999999999999,"5min":15.726000000000001,"15min":23.890999999999998},"min":{"1min":8.3209999999999997,"5min":5.5179999999999998,"15min":5.5179999999999998},"max":{"1min":18.693999999999999,"5min":292.69600000000003,"15min":292.69600000000003},"last":12.515000000000001},{"interface":"front","average":{"1min":10.73,"5min":17.122,"15min":33.595999999999997},"min":{"1min":7.3499999999999996,"5min":6.9059999999999997,"15min":6.9059999999999997},"max":{"1min":19.689,"5min":571.03399999999999,"15min":571.03399999999999},"last":11.326000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:52:56 2023","interfaces":[{"interface":"back","average":{"1min":16.391999999999999,"5min":16.396999999999998,"15min":18.145},"min":{"1min":11.743,"5min":8.3689999999999998,"15min":8.3689999999999998},"max":{"1min":22.370000000000001,"5min":62.984000000000002,"15min":62.984000000000002},"last":13.478},{"interface":"front","average":{"1min":13.821999999999999,"5min":14.734,"15min":16.702000000000002},"min":{"1min":8.3650000000000002,"5min":7.875,"15min":7.875},"max":{"1min":21.393000000000001,"5min":59.280999999999999,"15min":59.280999999999999},"last":16.472999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":13.872999999999999,"5min":16.023,"15min":17.783000000000001},"min":{"1min":9.3239999999999998,"5min":7.8949999999999996,"15min":7.8949999999999996},"max":{"1min":21.843,"5min":65.084000000000003,"15min":65.084000000000003},"last":15.776999999999999},{"interface":"front","average":{"1min":16.149999999999999,"5min":14.801,"15min":16.591000000000001},"min":{"1min":8.1530000000000005,"5min":6.3650000000000002,"15min":6.3650000000000002},"max":{"1min":21.564,"5min":56.024999999999999,"15min":56.024999999999999},"last":19.713000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":14.815,"5min":15.505000000000001,"15min":16.719000000000001},"min":{"1min":9.6229999999999993,"5min":6.5369999999999999,"15min":6.5369999999999999},"max":{"1min":20.928000000000001,"5min":39.575000000000003,"15min":39.575000000000003},"last":17.234000000000002},{"interface":"front","average":{"1min":16.652999999999999,"5min":18.010000000000002,"15min":20.782},"min":{"1min":9.0199999999999996,"5min":8.5459999999999994,"15min":8.5459999999999994},"max":{"1min":20.765999999999998,"5min":113.193,"15min":113.193},"last":14.387}]},{"osd":5,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":18.452000000000002,"5min":18.542000000000002,"15min":19.276},"min":{"1min":15.332000000000001,"5min":10.16,"15min":10.16},"max":{"1min":22.913,"5min":49.593000000000004,"15min":49.593000000000004},"last":20.428999999999998},{"interface":"front","average":{"1min":16.960999999999999,"5min":18.082000000000001,"15min":19.195},"min":{"1min":12.218999999999999,"5min":7.6820000000000004,"15min":7.6820000000000004},"max":{"1min":20.646000000000001,"5min":48.204000000000001,"15min":48.204000000000001},"last":15.061}]},{"osd":6,"last update":"Sat Sep 30 00:52:53 2023","interfaces":[{"interface":"back","average":{"1min":18.614999999999998,"5min":19.957999999999998,"15min":23.268000000000001},"min":{"1min":14.460000000000001,"5min":10.625999999999999,"15min":10.625999999999999},"max":{"1min":23.527000000000001,"5min":115.434,"15min":115.434},"last":21.224},{"interface":"front","average":{"1min":17.231999999999999,"5min":16.93,"15min":17.43},"min":{"1min":12.894,"5min":8.0190000000000001,"15min":8.0190000000000001},"max":{"1min":22.411999999999999,"5min":40.805,"15min":40.805},"last":19.001000000000001}]}]},{"osd":0,"up_from":20,"seq":85899345995,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":73459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":12.696999999999999,"5min":13.567,"15min":14.483000000000001},"min":{"1min":5.0830000000000002,"5min":5.0830000000000002,"15min":5.0830000000000002},"max":{"1min":20.931000000000001,"5min":29.138999999999999,"15min":29.138999999999999},"last":12.157999999999999},{"interface":"front","average":{"1min":11.182,"5min":12.558999999999999,"15min":13.718999999999999},"min":{"1min":6.9989999999999997,"5min":6.1059999999999999,"15min":6.1059999999999999},"max":{"1min":20.055,"5min":35.32,"15min":35.32},"last":11.465999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":9.7750000000000004,"5min":11.563000000000001,"15min":14.34},"min":{"1min":7.3109999999999999,"5min":6.9829999999999997,"15min":6.9829999999999997},"max":{"1min":14.356,"5min":56.628999999999998,"15min":56.628999999999998},"last":17.280000000000001},{"interface":"front","average":{"1min":16.048999999999999,"5min":17.128,"15min":19.702999999999999},"min":{"1min":11.364000000000001,"5min":10.457000000000001,"15min":10.457000000000001},"max":{"1min":20.681000000000001,"5min":50.646000000000001,"15min":50.646000000000001},"last":16.561}]},{"osd":3,"last update":"Sat Sep 30 00:53:06 2023","interfaces":[{"interface":"back","average":{"1min":14.436,"5min":21.872,"15min":41.698},"min":{"1min":9.6940000000000008,"5min":8.4399999999999995,"15min":8.4399999999999995},"max":{"1min":20.186,"5min":616.83500000000004,"15min":616.83500000000004},"last":7.0999999999999996},{"interface":"front","average":{"1min":14.864000000000001,"5min":21.122,"15min":36.988999999999997},"min":{"1min":8.2059999999999995,"5min":8.2059999999999995,"15min":8.2059999999999995},"max":{"1min":20.460000000000001,"5min":495.32600000000002,"15min":495.32600000000002},"last":10.451000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":17.096,"5min":17.635999999999999,"15min":19.018999999999998},"min":{"1min":11.279999999999999,"5min":9.875,"15min":9.875},"max":{"1min":23.553999999999998,"5min":36.109000000000002,"15min":36.109000000000002},"last":12.954000000000001},{"interface":"front","average":{"1min":15.769,"5min":16.484999999999999,"15min":18.593},"min":{"1min":10.702,"5min":8.4420000000000002,"15min":8.4420000000000002},"max":{"1min":22.722999999999999,"5min":39.177999999999997,"15min":39.177999999999997},"last":13.651999999999999}]},{"osd":5,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":18.515000000000001,"5min":19.923999999999999,"15min":22.416},"min":{"1min":12.333,"5min":9.3870000000000005,"15min":9.3870000000000005},"max":{"1min":26.995999999999999,"5min":66.313999999999993,"15min":66.313999999999993},"last":15.494999999999999},{"interface":"front","average":{"1min":20.390999999999998,"5min":22.050999999999998,"15min":23.927},"min":{"1min":12.085000000000001,"5min":12.085000000000001,"15min":12.085000000000001},"max":{"1min":25.503,"5min":41.634,"15min":41.634},"last":18.550999999999998}]},{"osd":6,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":14.702999999999999,"5min":17.117999999999999,"15min":22.646000000000001},"min":{"1min":10.412000000000001,"5min":8.1270000000000007,"15min":8.1270000000000007},"max":{"1min":21.728000000000002,"5min":54.033999999999999,"15min":54.033999999999999},"last":19.32},{"interface":"front","average":{"1min":20.824999999999999,"5min":22.52,"15min":24.59},"min":{"1min":16.77,"5min":11.055999999999999,"15min":11.055999999999999},"max":{"1min":24.806000000000001,"5min":69.688000000000002,"15min":69.688000000000002},"last":20.887}]},{"osd":7,"last update":"Sat Sep 30 00:52:57 2023","interfaces":[{"interface":"back","average":{"1min":22.016999999999999,"5min":23.597999999999999,"15min":26.277000000000001},"min":{"1min":16.298999999999999,"5min":14.875,"15min":14.875},"max":{"1min":27.872,"5min":49.347999999999999,"15min":49.347999999999999},"last":22.768999999999998},{"interface":"front","average":{"1min":16.75,"5min":18.710999999999999,"15min":21.469000000000001},"min":{"1min":10.622,"5min":10.489000000000001,"15min":10.489000000000001},"max":{"1min":24.265000000000001,"5min":54.734999999999999,"15min":54.734999999999999},"last":20.109000000000002}]}]},{"osd":1,"up_from":19,"seq":81604378698,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5856,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365984,"statfs":{"total":96636764160,"available":96630767616,"internally_reserved":0,"allocated":614400,"data_stored":471299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":10.407999999999999,"5min":13.113,"15min":16.02},"min":{"1min":7.4809999999999999,"5min":7.0110000000000001,"15min":7.0110000000000001},"max":{"1min":13.616,"5min":91.582999999999998,"15min":91.582999999999998},"last":17.053000000000001},{"interface":"front","average":{"1min":12.308,"5min":13.007999999999999,"15min":14.891999999999999},"min":{"1min":8.1780000000000008,"5min":7.2889999999999997,"15min":7.2889999999999997},"max":{"1min":26.073,"5min":75.853999999999999,"15min":75.853999999999999},"last":9.0619999999999994}]},{"osd":2,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":12.657999999999999,"5min":14.127000000000001,"15min":18.731000000000002},"min":{"1min":8.2639999999999993,"5min":8.2639999999999993,"15min":8.2639999999999993},"max":{"1min":20.702000000000002,"5min":142.41999999999999,"15min":142.41999999999999},"last":9.5730000000000004},{"interface":"front","average":{"1min":15.305,"5min":16.303999999999998,"15min":19.814},"min":{"1min":9.952,"5min":8.0749999999999993,"15min":8.0749999999999993},"max":{"1min":23.823,"5min":133.988,"15min":133.988},"last":10.933}]},{"osd":3,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":14.644,"5min":17.609000000000002,"15min":23.707000000000001},"min":{"1min":8.9009999999999998,"5min":8.4019999999999992,"15min":8.4019999999999992},"max":{"1min":21.539000000000001,"5min":170.785,"15min":170.785},"last":21.805},{"interface":"front","average":{"1min":16.152000000000001,"5min":19.57,"15min":27.786000000000001},"min":{"1min":9.1349999999999998,"5min":9.1349999999999998,"15min":9.1349999999999998},"max":{"1min":23.108000000000001,"5min":170.17400000000001,"15min":170.17400000000001},"last":15.239000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":15.271000000000001,"5min":18.404,"15min":24.379000000000001},"min":{"1min":10.272,"5min":8.9510000000000005,"15min":8.9510000000000005},"max":{"1min":19.027999999999999,"5min":108.045,"15min":108.045},"last":23.370000000000001},{"interface":"front","average":{"1min":17.634,"5min":20.315000000000001,"15min":27.440999999999999},"min":{"1min":10.829000000000001,"5min":9.9000000000000004,"15min":9.9000000000000004},"max":{"1min":23.558,"5min":141.97300000000001,"15min":141.97300000000001},"last":14.538}]},{"osd":5,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":16.079000000000001,"5min":20.795999999999999,"15min":26.815999999999999},"min":{"1min":9.9489999999999998,"5min":9.9489999999999998,"15min":9.9489999999999998},"max":{"1min":22.390000000000001,"5min":116.873,"15min":116.873},"last":25.155999999999999},{"interface":"front","average":{"1min":19.495000000000001,"5min":22.710999999999999,"15min":31.870999999999999},"min":{"1min":11.393000000000001,"5min":9.9550000000000001,"15min":9.9550000000000001},"max":{"1min":24.968,"5min":199.773,"15min":199.773},"last":18.082000000000001}]},{"osd":6,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":19.128,"5min":22.716000000000001,"15min":28.771999999999998},"min":{"1min":12.276999999999999,"5min":10.601000000000001,"15min":10.601000000000001},"max":{"1min":23.670999999999999,"5min":131.06299999999999,"15min":131.06299999999999},"last":26.247},{"interface":"front","average":{"1min":20.283000000000001,"5min":23.718,"15min":31.068000000000001},"min":{"1min":14.736000000000001,"5min":13.375999999999999,"15min":13.375999999999999},"max":{"1min":27.178000000000001,"5min":139.024,"15min":139.024},"last":22.684000000000001}]},{"osd":7,"last update":"Sat Sep 30 00:52:51 2023","interfaces":[{"interface":"back","average":{"1min":16.785,"5min":19.227,"15min":26.094999999999999},"min":{"1min":9.577,"5min":8.0589999999999993,"15min":8.0589999999999993},"max":{"1min":27.773,"5min":127.039,"15min":127.039},"last":11.981},{"interface":"front","average":{"1min":18.727,"5min":21.361999999999998,"15min":27.317},"min":{"1min":11.532,"5min":10.449,"15min":10.449},"max":{"1min":24.533999999999999,"5min":130.88499999999999,"15min":130.88499999999999},"last":12.92}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:53:38.806 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-30T00:53:38.936 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.936+0000 7f93fe2ec700 10 monclient: get_monmap_and_config 2023-09-30T00:53:38.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.936+0000 7f93fe2ec700 10 monclient: build_initial_monmap 2023-09-30T00:53:38.937 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.936+0000 7f93fe2ec700 10 monclient: monmap: 2023-09-30T00:53:38.937 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:38.937 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:38.938 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:53:38.937298+0000 2023-09-30T00:53:38.938 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:53:38.937298+0000 2023-09-30T00:53:38.938 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:38.938 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:38.938 INFO:teuthology.orchestra.run.smithi040.stderr:0: v1:172.21.15.40:6789/0 mon.noname-a-legacy 2023-09-30T00:53:38.938 INFO:teuthology.orchestra.run.smithi040.stderr:1: v1:172.21.15.135:6789/0 mon.noname-c-legacy 2023-09-30T00:53:38.938 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-b 2023-09-30T00:53:38.938 INFO:teuthology.orchestra.run.smithi040.stderr:3: v2:172.21.15.40:3300/0 mon.noname-a 2023-09-30T00:53:38.938 INFO:teuthology.orchestra.run.smithi040.stderr:4: v2:172.21.15.135:3300/0 mon.noname-c 2023-09-30T00:53:38.938 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:38.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 1 Processor -- start 2023-09-30T00:53:38.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 1 -- start start 2023-09-30T00:53:38.940 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 10 monclient: init 2023-09-30T00:53:38.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:38.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 10 monclient: _add_conns ranks=[2,1,3,0,4] 2023-09-30T00:53:38.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93f80b1460 0x7f93f80b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:38.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 10 monclient(hunting): picked mon.noname-b con 0x7f93f80b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:38.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93f80a5fb0 addr v1:172.21.15.135:6789/0 2023-09-30T00:53:38.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f93f813ac20 0x7f93f813b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:38.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 10 monclient(hunting): picked mon.noname-a con 0x7f93f813ac20 addr v2:172.21.15.40:3300/0 2023-09-30T00:53:38.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f80a5b20 con 0x7f93f80a5fb0 2023-09-30T00:53:38.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:38.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93f80a71f0 con 0x7f93f813ac20 2023-09-30T00:53:38.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:38.941 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f80a74d0 con 0x7f93f80b1460 2023-09-30T00:53:38.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:38.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.937+0000 7f93fe2ec700 10 monclient(hunting): authenticate will time out at 5680.238775s 2023-09-30T00:53:38.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.938+0000 7f93fc889700 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f93f813ac20 0x7f93f813b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:38.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.938+0000 7f93f7fff700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93f80b1460 0x7f93f80b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:38.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.938+0000 7f93f77fe700 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f93f80a5fb0 0x7f93f8141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.40:51988/0 (socket says 172.21.15.40:51988) 2023-09-30T00:53:38.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.938+0000 7f93f77fe700 1 -- 172.21.15.40:0/3041982835 learned_addr learned my addr 172.21.15.40:0/3041982835 (peer_addr_for_me v1:172.21.15.40:0/0) 2023-09-30T00:53:38.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.938+0000 7f93fc889700 10 monclient(hunting): get_auth_request con 0x7f93f813ac20 auth_method 0 2023-09-30T00:53:38.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.938+0000 7f93fc889700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:38.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.938+0000 7f93fc889700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:38.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.938+0000 7f93fc889700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:38.942 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.938+0000 7f93f7fff700 10 monclient(hunting): get_auth_request con 0x7f93f80b1460 auth_method 0 2023-09-30T00:53:38.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.938+0000 7f93f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:38.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.938+0000 7f93f7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:38.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.938+0000 7f93f7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:38.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.939+0000 7f93fc889700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:38.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.939+0000 7f93fc889700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:38.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.939+0000 7f93fc889700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:38.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.939+0000 7f93f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:38.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.939+0000 7f93f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:38.943 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.939+0000 7f93f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:38.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.941+0000 7f93f6ffd700 1 -- 172.21.15.40:0/3041982835 <== mon.1 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656904486 0 0) 0x7f93f80a5b20 con 0x7f93f80a5fb0 2023-09-30T00:53:38.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.941+0000 7f93f6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:38.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.941+0000 7f93f6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:38.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.941+0000 7f93f6ffd700 10 monclient(hunting): my global_id is 5011 2023-09-30T00:53:38.944 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.941+0000 7f93f6ffd700 1 -- 172.21.15.40:0/3041982835 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e0003570 con 0x7f93f80a5fb0 2023-09-30T00:53:38.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.941+0000 7f93fc889700 10 monclient(hunting): handle_auth_done global_id 4914 payload 274 2023-09-30T00:53:38.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.941+0000 7f93fc889700 1 -- 172.21.15.40:0/3041982835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93f80b1460 msgr2=0x7f93f80b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:53:38.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.941+0000 7f93fc889700 1 --2- 172.21.15.40:0/3041982835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93f80b1460 0x7f93f80b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.941+0000 7f93fc889700 1 -- 172.21.15.40:0/3041982835 >> v1:172.21.15.135:6789/0 conn(0x7f93f80a5fb0 legacy=0x7f93f8141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:38.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.941+0000 7f93fc889700 10 monclient: _finish_hunting 0 2023-09-30T00:53:38.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.941+0000 7f93fc889700 1 monclient: found mon.noname-a 2023-09-30T00:53:38.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.941+0000 7f93fc889700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:38.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.941+0000 7f93fc889700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.40:3300/0 2023-09-30T00:53:38.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.941+0000 7f93fc889700 1 -- 172.21.15.40:0/3041982835 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f80a63c0 con 0x7f93f813ac20 2023-09-30T00:53:38.945 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.943+0000 7f93fc889700 1 --2- 172.21.15.40:0/3041982835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93f813ac20 0x7f93f813b0a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f93f0009e60 tx=0x7f93f0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538ef68acf7dc17d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:38.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93f6ffd700 1 -- 172.21.15.40:0/3041982835 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f000e070 con 0x7f93f813ac20 2023-09-30T00:53:38.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:38.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93f6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:53:38.949 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93f6ffd700 10 monclient: dump: 2023-09-30T00:53:38.949 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:38.949 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:38.949 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.950 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.950 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:38.950 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:38.950 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:38.950 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:38.950 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:38.950 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:38.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93f6ffd700 10 monclient: _finish_auth 0 2023-09-30T00:53:38.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93f6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:38.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93f6ffd700 1 -- 172.21.15.40:0/3041982835 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f000bab0 con 0x7f93f813ac20 2023-09-30T00:53:38.950 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93f6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:38.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93f6ffd700 1 -- 172.21.15.40:0/3041982835 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f000be70 con 0x7f93f813ac20 2023-09-30T00:53:38.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:38.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93f6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:53:38.951 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93f6ffd700 10 monclient: dump: 2023-09-30T00:53:38.951 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:38.951 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:38.951 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.951 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.951 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:38.951 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:38.951 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:38.951 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:38.951 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:38.952 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:38.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93fe2ec700 5 monclient: authenticate success, global_id 4914 2023-09-30T00:53:38.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93fe2ec700 10 monclient: get_monmap_and_config success 2023-09-30T00:53:38.952 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.948+0000 7f93fe2ec700 4 set_mon_vals no callback set 2023-09-30T00:53:38.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.949+0000 7f93fe2ec700 10 monclient: shutdown 2023-09-30T00:53:38.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.949+0000 7f93fe2ec700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:38.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.949+0000 7f93fe2ec700 1 -- 172.21.15.40:0/3041982835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93f813ac20 msgr2=0x7f93f813b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:38.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.949+0000 7f93fe2ec700 1 --2- 172.21.15.40:0/3041982835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93f813ac20 0x7f93f813b0a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f93f0009e60 tx=0x7f93f0009780 comp rx=0 tx=0).stop 2023-09-30T00:53:38.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.949+0000 7f93fe2ec700 1 -- 172.21.15.40:0/3041982835 shutdown_connections 2023-09-30T00:53:38.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.949+0000 7f93fe2ec700 1 --2- 172.21.15.40:0/3041982835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93f813ac20 0x7f93f813b0a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.949+0000 7f93fe2ec700 1 --2- 172.21.15.40:0/3041982835 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93f80b1460 0x7f93f80b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.955 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.949+0000 7f93fe2ec700 1 -- 172.21.15.40:0/3041982835 >> 172.21.15.40:0/3041982835 conn(0x7f93f80631f0 msgr2=0x7f93f80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:38.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.949+0000 7f93fe2ec700 1 -- 172.21.15.40:0/3041982835 shutdown_connections 2023-09-30T00:53:38.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.949+0000 7f93fe2ec700 1 -- 172.21.15.40:0/3041982835 wait complete. 2023-09-30T00:53:38.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.950+0000 7f93fe2ec700 10 monclient: build_initial_monmap 2023-09-30T00:53:38.956 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.950+0000 7f93fe2ec700 10 monclient: monmap: 2023-09-30T00:53:38.956 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 0 2023-09-30T00:53:38.956 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:53:38.956 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 0.000000 2023-09-30T00:53:38.956 INFO:teuthology.orchestra.run.smithi040.stderr:created 0.000000 2023-09-30T00:53:38.956 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 0 (unknown) 2023-09-30T00:53:38.956 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 1 2023-09-30T00:53:38.956 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.noname-a 2023-09-30T00:53:38.956 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.noname-b 2023-09-30T00:53:38.957 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.noname-c 2023-09-30T00:53:38.957 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:38.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.950+0000 7f93fe2ec700 1 librados: starting msgr at 2023-09-30T00:53:38.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.950+0000 7f93fe2ec700 1 librados: starting objecter 2023-09-30T00:53:38.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.950+0000 7f93fe2ec700 1 Processor -- start 2023-09-30T00:53:38.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 1 -- start start 2023-09-30T00:53:38.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 1 librados: setting wanted keys 2023-09-30T00:53:38.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 1 librados: calling monclient init 2023-09-30T00:53:38.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 10 monclient: init 2023-09-30T00:53:38.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 10 monclient: _reopen_session rank -1 2023-09-30T00:53:38.957 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:53:38.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93f80a5fb0 0x7f93f80ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:38.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 10 monclient(hunting): picked mon.noname-b con 0x7f93f80a5fb0 addr [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] 2023-09-30T00:53:38.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93f80b1460 0x7f93f80ac520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:38.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 10 monclient(hunting): picked mon.noname-c con 0x7f93f80b1460 addr [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] 2023-09-30T00:53:38.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93f813ac20 0x7f93f80a9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:38.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 10 monclient(hunting): picked mon.noname-a con 0x7f93f813ac20 addr [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] 2023-09-30T00:53:38.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:38.958 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f80ae6c0 con 0x7f93f813ac20 2023-09-30T00:53:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 1 -- --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f80ae930 con 0x7f93f80a5fb0 2023-09-30T00:53:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:53:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 1 -- --> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93f80ab690 con 0x7f93f80b1460 2023-09-30T00:53:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 10 monclient(hunting): _renew_subs 2023-09-30T00:53:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.951+0000 7f93fe2ec700 10 monclient(hunting): authenticate will time out at 5680.252825s 2023-09-30T00:53:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.952+0000 7f93fc889700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93f813ac20 0x7f93f80a9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.952+0000 7f93f7fff700 1 --2- >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93f80a5fb0 0x7f93f80ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.952+0000 7f93f77fe700 1 --2- >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93f80b1460 0x7f93f80ac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.952+0000 7f93fc889700 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93f813ac20 0x7f93f80a9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34794/0 (socket says 172.21.15.40:34794) 2023-09-30T00:53:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.952+0000 7f93fc889700 1 -- 172.21.15.40:0/2377464166 learned_addr learned my addr 172.21.15.40:0/2377464166 (peer_addr_for_me v2:172.21.15.40:0/0) 2023-09-30T00:53:38.959 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.952+0000 7f93fc889700 10 monclient(hunting): get_auth_request con 0x7f93f813ac20 auth_method 0 2023-09-30T00:53:38.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.952+0000 7f93fc889700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:38.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.952+0000 7f93fc889700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:38.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.952+0000 7f93fc889700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:38.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.952+0000 7f93f7fff700 10 monclient(hunting): get_auth_request con 0x7f93f80a5fb0 auth_method 0 2023-09-30T00:53:38.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.952+0000 7f93f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:38.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.952+0000 7f93f7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:38.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.952+0000 7f93f7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:38.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.953+0000 7f93f77fe700 10 monclient(hunting): get_auth_request con 0x7f93f80b1460 auth_method 0 2023-09-30T00:53:38.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.953+0000 7f93f77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:53:38.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.953+0000 7f93f77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:53:38.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.953+0000 7f93f77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:53:38.960 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.953+0000 7f93f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:38.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.953+0000 7f93f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:38.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.953+0000 7f93f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:38.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.953+0000 7f93fc889700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:38.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.953+0000 7f93fc889700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:38.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.953+0000 7f93fc889700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:38.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.953+0000 7f93f77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:53:38.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.953+0000 7f93f77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:53:38.961 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.953+0000 7f93f77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:53:38.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.956+0000 7f93f7fff700 10 monclient(hunting): handle_auth_done global_id 5014 payload 824 2023-09-30T00:53:38.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.956+0000 7f93f7fff700 1 -- 172.21.15.40:0/2377464166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93f80b1460 msgr2=0x7f93f80ac520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:38.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.956+0000 7f93f7fff700 1 --2- 172.21.15.40:0/2377464166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93f80b1460 0x7f93f80ac520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.956+0000 7f93f7fff700 1 -- 172.21.15.40:0/2377464166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93f813ac20 msgr2=0x7f93f80a9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:38.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.956+0000 7f93f7fff700 1 --2- 172.21.15.40:0/2377464166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93f813ac20 0x7f93f80a9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:38.963 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.956+0000 7f93f7fff700 10 monclient: _finish_hunting 0 2023-09-30T00:53:38.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.956+0000 7f93f7fff700 1 monclient: found mon.noname-b 2023-09-30T00:53:38.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.956+0000 7f93f7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:53:38.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.956+0000 7f93f7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.135:3300/0 2023-09-30T00:53:38.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.956+0000 7f93f7fff700 1 -- 172.21.15.40:0/2377464166 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f0009a20 con 0x7f93f80a5fb0 2023-09-30T00:53:38.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.958+0000 7f93f7fff700 1 --2- 172.21.15.40:0/2377464166 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93f80a5fb0 0x7f93f80ad2d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f93e800ea50 tx=0x7f93e80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:38.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.958+0000 7f93dffff700 10 client.?.objecter ms_handle_connect 0x7f93f80a5fb0 2023-09-30T00:53:38.964 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.958+0000 7f93dffff700 10 client.?.objecter resend_mon_ops 2023-09-30T00:53:38.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93dffff700 1 -- 172.21.15.40:0/2377464166 <== mon.1 v2:172.21.15.135:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e800ee90 con 0x7f93f80a5fb0 2023-09-30T00:53:38.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93dffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:38.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93dffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:53:38.965 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93dffff700 10 monclient: dump: 2023-09-30T00:53:38.965 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:38.965 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:38.965 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.965 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.965 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:38.965 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:38.965 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:38.965 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:38.966 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:38.966 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:38.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93dffff700 10 monclient: _finish_auth 0 2023-09-30T00:53:38.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93dffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:53:38.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93dffff700 1 -- 172.21.15.40:0/2377464166 <== mon.1 v2:172.21.15.135:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e800c7f0 con 0x7f93f80a5fb0 2023-09-30T00:53:38.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93dffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:53:38.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93dffff700 1 -- 172.21.15.40:0/2377464166 <== mon.1 v2:172.21.15.135:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e800cb90 con 0x7f93f80a5fb0 2023-09-30T00:53:38.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93dffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:53:38.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93dffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:53:38.966 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93dffff700 10 monclient: dump: 2023-09-30T00:53:38.966 INFO:teuthology.orchestra.run.smithi040.stderr:epoch 1 2023-09-30T00:53:38.967 INFO:teuthology.orchestra.run.smithi040.stderr:fsid 238dba7c-9190-48bd-bb67-322ebc9ecbd0 2023-09-30T00:53:38.967 INFO:teuthology.orchestra.run.smithi040.stderr:last_changed 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.967 INFO:teuthology.orchestra.run.smithi040.stderr:created 2023-09-30T00:45:51.366050+0000 2023-09-30T00:53:38.967 INFO:teuthology.orchestra.run.smithi040.stderr:min_mon_release 17 (quincy) 2023-09-30T00:53:38.967 INFO:teuthology.orchestra.run.smithi040.stderr:election_strategy: 3 2023-09-30T00:53:38.967 INFO:teuthology.orchestra.run.smithi040.stderr:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2023-09-30T00:53:38.967 INFO:teuthology.orchestra.run.smithi040.stderr:1: [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] mon.b 2023-09-30T00:53:38.967 INFO:teuthology.orchestra.run.smithi040.stderr:2: [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] mon.c 2023-09-30T00:53:38.967 INFO:teuthology.orchestra.run.smithi040.stderr: 2023-09-30T00:53:38.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93f67fc700 4 set_mon_vals no callback set 2023-09-30T00:53:38.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93fe2ec700 5 monclient: authenticate success, global_id 5014 2023-09-30T00:53:38.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93fe2ec700 10 monclient: _renew_subs 2023-09-30T00:53:38.967 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93fe2ec700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:38.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93fe2ec700 1 -- 172.21.15.40:0/2377464166 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f80aa200 con 0x7f93f80a5fb0 2023-09-30T00:53:38.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93fe2ec700 10 client.5014.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T00:53:38.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93fe2ec700 10 monclient: _renew_subs 2023-09-30T00:53:38.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93fe2ec700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:38.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93fe2ec700 1 -- 172.21.15.40:0/2377464166 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f80aa720 con 0x7f93f80a5fb0 2023-09-30T00:53:38.968 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.962+0000 7f93fe2ec700 1 librados: init done 2023-09-30T00:53:38.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.963+0000 7f93fe2ec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:38.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.963+0000 7f93fe2ec700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:38.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.963+0000 7f93fe2ec700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:38.969 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:38.963+0000 7f93fe2ec700 1 -- 172.21.15.40:0/2377464166 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f80aa9a0 con 0x7f93f80a5fb0 2023-09-30T00:53:39.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.039+0000 7f93dffff700 1 -- 172.21.15.40:0/2377464166 <== mon.1 v2:172.21.15.135:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f93e8013470 con 0x7f93f80a5fb0 2023-09-30T00:53:39.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.039+0000 7f93dffff700 1 --2- 172.21.15.40:0/2377464166 >> v2:172.21.15.135:6800/116426 conn(0x7f93e4068960 0x7f93e406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:53:39.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.040+0000 7f93f77fe700 1 --2- 172.21.15.40:0/2377464166 >> v2:172.21.15.135:6800/116426 conn(0x7f93e4068960 0x7f93e406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:53:39.040 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.040+0000 7f93f77fe700 10 monclient: get_auth_request con 0x7f93e4068960 auth_method 0 2023-09-30T00:53:39.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.040+0000 7f93f77fe700 1 --2- 172.21.15.40:0/2377464166 >> v2:172.21.15.135:6800/116426 conn(0x7f93e4068960 0x7f93e406ae10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f93f80ae2a0 tx=0x7f93ec009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:53:39.041 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.040+0000 7f93dffff700 10 client.5014.objecter ms_handle_connect 0x7f93e4068960 2023-09-30T00:53:39.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.213+0000 7f93dffff700 1 -- 172.21.15.40:0/2377464166 <== mon.1 v2:172.21.15.135:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f93e80564c0 con 0x7f93f80a5fb0 2023-09-30T00:53:39.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.213+0000 7f93dffff700 10 client.5014.objecter ms_dispatch 0x7f93f8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T00:53:39.213 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.213+0000 7f93dffff700 3 client.5014.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T00:53:39.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.213+0000 7f93dffff700 3 client.5014.objecter handle_osd_map decoding full epoch 25 2023-09-30T00:53:39.214 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.213+0000 7f93dffff700 20 client.5014.objecter dump_active .. 0 homeless 2023-09-30T00:53:39.222 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.222+0000 7f93dffff700 1 -- 172.21.15.40:0/2377464166 <== mon.1 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f93e8052100 con 0x7f93f80a5fb0 2023-09-30T00:53:39.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.222+0000 7f93dffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:53:39.223 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.222+0000 7f93dffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:53:39.416 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:07:15.127 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:39.419 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:07:15.131 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:39.454 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:07:15.165 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:39.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.465+0000 7f93fe2ec700 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T00:53:39.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.465+0000 7f93fe2ec700 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T00:53:39.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.465+0000 7f93fe2ec700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.135:3300/0 2023-09-30T00:53:39.465 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:39.465+0000 7f93fe2ec700 1 -- 172.21.15.40:0/2377464166 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f93f80aea70 con 0x7f93f80a5fb0 2023-09-30T00:53:39.472 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:15.159 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:39.480 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:15.167 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:39.480 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:07:15.192 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:39.489 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:07:15.201 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:39.501 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:15.188 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:39.513 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:07:15.225 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:40.034 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.034+0000 7f93dffff700 1 -- 172.21.15.40:0/2377464166 <== mon.1 v2:172.21.15.135:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f93e8055e40 con 0x7f93f80a5fb0 2023-09-30T00:53:40.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.034+0000 7f93dffff700 10 monclient: handle_config config(1 keys) v1 2023-09-30T00:53:40.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.034+0000 7f93f67fc700 4 set_mon_vals no callback set 2023-09-30T00:53:40.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.034+0000 7f93f67fc700 10 set_mon_vals mon_health_to_clog = false 2023-09-30T00:53:40.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.034+0000 7f93dffff700 1 -- 172.21.15.40:0/2377464166 <== mon.1 v2:172.21.15.135:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f93f80aea70 con 0x7f93f80a5fb0 2023-09-30T00:53:40.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.034+0000 7f93dffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T00:53:40.035 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.034+0000 7f93dffff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:53:40.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 10 librados: watch_flush enter 2023-09-30T00:53:40.037 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 10 librados: watch_flush exit 2023-09-30T00:53:40.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 20 client.5014.objecter shutdown clearing up homeless session... 2023-09-30T00:53:40.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 10 client.5014.objecter successfully canceled tick 2023-09-30T00:53:40.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 1 -- 172.21.15.40:0/2377464166 >> v2:172.21.15.135:6800/116426 conn(0x7f93e4068960 msgr2=0x7f93e406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:40.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 1 --2- 172.21.15.40:0/2377464166 >> v2:172.21.15.135:6800/116426 conn(0x7f93e4068960 0x7f93e406ae10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f93f80ae2a0 tx=0x7f93ec009040 comp rx=0 tx=0).stop 2023-09-30T00:53:40.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 10 monclient: shutdown 2023-09-30T00:53:40.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:53:40.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 1 -- 172.21.15.40:0/2377464166 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93f80a5fb0 msgr2=0x7f93f80ad2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:53:40.038 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 1 --2- 172.21.15.40:0/2377464166 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93f80a5fb0 0x7f93f80ad2d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f93e800ea50 tx=0x7f93e80091a0 comp rx=0 tx=0).stop 2023-09-30T00:53:40.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 1 -- 172.21.15.40:0/2377464166 shutdown_connections 2023-09-30T00:53:40.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 1 --2- 172.21.15.40:0/2377464166 >> v2:172.21.15.135:6800/116426 conn(0x7f93e4068960 0x7f93e406ae10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:40.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 1 --2- 172.21.15.40:0/2377464166 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f93f80a5fb0 0x7f93f80ad2d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:40.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 1 --2- 172.21.15.40:0/2377464166 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f93f813ac20 0x7f93f80a9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:40.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 1 --2- 172.21.15.40:0/2377464166 >> [v2:172.21.15.40:3301/0,v1:172.21.15.40:6790/0] conn(0x7f93f80b1460 0x7f93f80ac520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:53:40.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.036+0000 7f93fe2ec700 1 -- 172.21.15.40:0/2377464166 >> 172.21.15.40:0/2377464166 conn(0x7f93f80631f0 msgr2=0x7f93f80b51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:53:40.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.037+0000 7f93fe2ec700 1 -- 172.21.15.40:0/2377464166 shutdown_connections 2023-09-30T00:53:40.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.037+0000 7f93fe2ec700 1 -- 172.21.15.40:0/2377464166 wait complete. 2023-09-30T00:53:40.039 INFO:teuthology.orchestra.run.smithi040.stderr:2023-09-30T00:53:40.037+0000 7f93fe2ec700 1 librados: shutdown 2023-09-30T00:53:40.053 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:53:40.055 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T00:53:40.055 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T00:53:40.055 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-30T00:53:40.055 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:53:40.082 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-09-30T00:53:40.072+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 117216) UID: 0 2023-09-30T00:53:40.082 INFO:tasks.ceph.osd.0.smithi040.stderr:2023-09-30T00:53:40.074+0000 ee8a700 -1 osd.0 25 *** Got signal Terminated *** 2023-09-30T00:53:46.156 INFO:tasks.ceph.osd.0:Stopped 2023-09-30T00:53:46.157 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-30T00:53:46.157 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:53:46.174 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-09-30T00:53:46.163+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 117218) UID: 0 2023-09-30T00:53:46.174 INFO:tasks.ceph.osd.1.smithi040.stderr:2023-09-30T00:53:46.165+0000 ee8a700 -1 osd.1 27 *** Got signal Terminated *** 2023-09-30T00:53:52.259 INFO:tasks.ceph.osd.1:Stopped 2023-09-30T00:53:52.259 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-30T00:53:52.259 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:53:52.274 INFO:tasks.ceph.osd.2.smithi040.stderr:2023-09-30T00:53:52.265+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 117228) UID: 0 2023-09-30T00:53:52.278 INFO:tasks.ceph.osd.2.smithi040.stderr:2023-09-30T00:53:52.268+0000 ee8a700 -1 osd.2 29 *** Got signal Terminated *** 2023-09-30T00:53:58.361 INFO:tasks.ceph.osd.2:Stopped 2023-09-30T00:53:58.361 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-30T00:53:58.361 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:53:58.377 INFO:tasks.ceph.osd.3.smithi040.stderr:2023-09-30T00:53:58.367+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 117230) UID: 0 2023-09-30T00:53:58.380 INFO:tasks.ceph.osd.3.smithi040.stderr:2023-09-30T00:53:58.369+0000 ee8a700 -1 osd.3 31 *** Got signal Terminated *** 2023-09-30T00:53:59.541 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:07:35.252 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:59.546 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:07:35.258 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:59.548 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:07:35.260 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:59.552 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:35.239 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:59.553 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:35.241 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:59.587 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:35.274 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:59.596 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:07:35.308 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:59.605 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:07:35.316 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:53:59.636 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:07:35.348 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:54:04.463 INFO:tasks.ceph.osd.3:Stopped 2023-09-30T00:54:04.463 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-09-30T00:54:04.464 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:54:04.478 INFO:tasks.ceph.osd.4.smithi135.stderr:2023-09-30T00:54:04.469+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 117121) UID: 0 2023-09-30T00:54:04.479 INFO:tasks.ceph.osd.4.smithi135.stderr:2023-09-30T00:54:04.472+0000 ee8a700 -1 osd.4 34 *** Got signal Terminated *** 2023-09-30T00:54:10.566 INFO:tasks.ceph.osd.4:Stopped 2023-09-30T00:54:10.566 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-09-30T00:54:10.566 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:54:10.588 INFO:tasks.ceph.osd.5.smithi135.stderr:2023-09-30T00:54:10.571+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 117122) UID: 0 2023-09-30T00:54:10.588 INFO:tasks.ceph.osd.5.smithi135.stderr:2023-09-30T00:54:10.573+0000 ee8a700 -1 osd.5 35 *** Got signal Terminated *** 2023-09-30T00:54:16.668 INFO:tasks.ceph.osd.5:Stopped 2023-09-30T00:54:16.669 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-09-30T00:54:16.669 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:54:16.685 INFO:tasks.ceph.osd.6.smithi135.stderr:2023-09-30T00:54:16.674+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 117124) UID: 0 2023-09-30T00:54:16.685 INFO:tasks.ceph.osd.6.smithi135.stderr:2023-09-30T00:54:16.676+0000 ee8a700 -1 osd.6 37 *** Got signal Terminated *** 2023-09-30T00:54:19.586 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:07:55.297 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:54:19.644 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:55.331 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:54:19.647 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:55.334 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:54:19.648 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:07:55.360 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:54:19.672 INFO:tasks.ceph.mon.b.smithi135.stderr:==00:00:07:55.359 116425== Warning: unimplemented fcntl command: 1036 2023-09-30T00:54:19.685 INFO:tasks.ceph.mon.a.smithi040.stderr:==00:00:07:55.396 116840== Warning: unimplemented fcntl command: 1036 2023-09-30T00:54:19.729 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:07:55.441 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:54:19.737 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:07:55.449 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:54:19.778 INFO:tasks.ceph.mon.c.smithi040.stderr:==00:00:07:55.490 116839== Warning: unimplemented fcntl command: 1036 2023-09-30T00:54:22.770 INFO:tasks.ceph.osd.6:Stopped 2023-09-30T00:54:22.771 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-09-30T00:54:22.771 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:54:22.787 INFO:tasks.ceph.osd.7.smithi135.stderr:2023-09-30T00:54:22.776+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 117128) UID: 0 2023-09-30T00:54:22.789 INFO:tasks.ceph.osd.7.smithi135.stderr:2023-09-30T00:54:22.779+0000 ee8a700 -1 osd.7 37 *** Got signal Terminated *** 2023-09-30T00:54:23.308 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-09-30T00:54:28.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-09-30T00:54:28.813 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-09-30T00:54:28.872 INFO:tasks.ceph.osd.7:Stopped 2023-09-30T00:54:28.872 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T00:54:28.873 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-09-30T00:54:28.873 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:54:28.898 INFO:tasks.ceph.mgr.y:Stopped 2023-09-30T00:54:28.899 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-30T00:54:28.899 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:54:28.902 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:54:28.900+0000 7feaf265b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 116423) UID: 0 2023-09-30T00:54:28.902 INFO:tasks.ceph.mgr.x.smithi135.stderr:2023-09-30T00:54:28.900+0000 7feaf265b700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-09-30T00:54:28.925 INFO:tasks.ceph.mgr.x:Stopped 2023-09-30T00:54:28.925 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T00:54:28.925 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-30T00:54:28.926 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:54:28.961 INFO:tasks.ceph.mon.a.smithi040.stderr:2023-09-30T00:54:28.946+0000 dc10700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 116786) UID: 0 2023-09-30T00:54:28.962 INFO:tasks.ceph.mon.a.smithi040.stderr:2023-09-30T00:54:28.948+0000 dc10700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-09-30T00:54:34.016 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-09-30T00:54:34.016 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-09-30T00:54:35.027 INFO:tasks.ceph.mon.a:Stopped 2023-09-30T00:54:35.028 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-30T00:54:35.028 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:54:35.051 INFO:tasks.ceph.mon.c.smithi040.stderr:2023-09-30T00:54:35.034+0000 dc10700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 116787) UID: 0 2023-09-30T00:54:35.055 INFO:tasks.ceph.mon.c.smithi040.stderr:2023-09-30T00:54:35.036+0000 dc10700 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-09-30T00:54:39.118 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-09-30T00:54:39.118 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-09-30T00:54:41.131 INFO:tasks.ceph.mon.c:Stopped 2023-09-30T00:54:41.131 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-30T00:54:41.131 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:54:41.156 INFO:tasks.ceph.mon.b.smithi135.stderr:2023-09-30T00:54:41.136+0000 dc10700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 116420) UID: 0 2023-09-30T00:54:41.159 INFO:tasks.ceph.mon.b.smithi135.stderr:2023-09-30T00:54:41.138+0000 dc10700 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2023-09-30T00:54:44.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-09-30T00:54:44.119 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-09-30T00:54:47.233 INFO:tasks.ceph.mon.b:Stopped 2023-09-30T00:54:47.233 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T00:54:47.233 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-09-30T00:54:47.282 INFO:teuthology.orchestra.run.smithi040.stdout:1696034833.9469364 mon.a (mon.0) 70 : cluster [ERR] Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T00:54:47.282 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-30T00:54:47.283 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-09-30T00:54:47.327 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-09-30T00:54:47.413 INFO:teuthology.orchestra.run.smithi040.stdout:1696034833.9469364 mon.a (mon.0) 70 : cluster [ERR] Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T00:54:47.413 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi040.front.sepia.ceph.com 2023-09-30T00:54:47.413 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-30T00:54:47.582 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi040.front.sepia.ceph.com 2023-09-30T00:54:47.583 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-30T00:54:47.670 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi040.front.sepia.ceph.com 2023-09-30T00:54:47.670 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-30T00:54:47.754 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi040.front.sepia.ceph.com 2023-09-30T00:54:47.754 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-30T00:54:47.812 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi135.front.sepia.ceph.com 2023-09-30T00:54:47.812 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-09-30T00:54:48.160 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi135.front.sepia.ceph.com 2023-09-30T00:54:48.161 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-09-30T00:54:48.228 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi135.front.sepia.ceph.com 2023-09-30T00:54:48.228 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-09-30T00:54:48.295 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi135.front.sepia.ceph.com 2023-09-30T00:54:48.295 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-09-30T00:54:48.365 INFO:tasks.ceph:Archiving mon data... 2023-09-30T00:54:48.366 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586/data/mon.a.tgz 2023-09-30T00:54:48.366 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2023-09-30T00:54:48.389 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.IBhDiLRNzS 2023-09-30T00:54:48.389 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.IBhDiLRNzS 2023-09-30T00:54:48.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.IBhDiLRNzS 2023-09-30T00:54:48.621 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.IBhDiLRNzS is 273KB 2023-09-30T00:54:48.719 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.IBhDiLRNzS 2023-09-30T00:54:48.742 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586/data/mon.c.tgz 2023-09-30T00:54:48.742 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2023-09-30T00:54:48.805 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.hMLVd9kNVi 2023-09-30T00:54:48.805 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.hMLVd9kNVi 2023-09-30T00:54:48.930 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.hMLVd9kNVi 2023-09-30T00:54:49.027 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.hMLVd9kNVi is 273KB 2023-09-30T00:54:49.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-09-30T00:54:49.124 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.hMLVd9kNVi 2023-09-30T00:54:49.148 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586/data/mon.b.tgz 2023-09-30T00:54:49.148 DEBUG:teuthology.orchestra.run.smithi135:> mktemp 2023-09-30T00:54:49.171 INFO:teuthology.orchestra.run.smithi135.stdout:/tmp/tmp.wFbIuI2SqL 2023-09-30T00:54:49.172 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.wFbIuI2SqL 2023-09-30T00:54:49.309 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0666 /tmp/tmp.wFbIuI2SqL 2023-09-30T00:54:49.399 DEBUG:teuthology.orchestra.remote:smithi135:/tmp/tmp.wFbIuI2SqL is 273KB 2023-09-30T00:54:49.493 DEBUG:teuthology.orchestra.run.smithi135:> rm -fr /tmp/tmp.wFbIuI2SqL 2023-09-30T00:54:49.512 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T00:54:49.512 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:54:49.514 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:54:49.579 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T00:54:49.579 DEBUG:teuthology.orchestra.run.smithi040:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:54:49.590 DEBUG:teuthology.orchestra.run.smithi135:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:54:49.663 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T00:54:49.665 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586/remote/smithi040/crash 2023-09-30T00:54:49.666 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:54:49.700 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586/remote/smithi135/crash 2023-09-30T00:54:49.701 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:54:49.733 INFO:tasks.ceph:Compressing logs... 2023-09-30T00:54:49.733 DEBUG:teuthology.orchestra.run.smithi040:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:54:49.745 DEBUG:teuthology.orchestra.run.smithi135:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:54:50.692 INFO:tasks.ceph:Archiving logs... 2023-09-30T00:54:50.692 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586/remote/smithi040/log 2023-09-30T00:54:50.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:54:50.988 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586/remote/smithi135/log 2023-09-30T00:54:50.989 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:54:51.130 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:54:51.203 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=64c9917d687e4242bccecdb58f4172b0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:54:51.205 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T00:54:51.230 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T00:54:51.246 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T00:54:51.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:54:51.260 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:54:51.343 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T00:54:51.352 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T00:54:51.352 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:54:51.356 DEBUG:teuthology.orchestra.run.smithi135:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:54:51.374 INFO:teuthology.orchestra.run.smithi135.stderr:bash: ntpq: command not found 2023-09-30T00:54:51.378 INFO:teuthology.orchestra.run.smithi040.stderr:bash: ntpq: command not found 2023-09-30T00:54:51.380 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:54:51.381 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2023-09-30T00:54:51.381 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 -288us[ -288us] +/- 91ms 2023-09-30T00:54:51.382 INFO:teuthology.orchestra.run.smithi135.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -341us[ -333us] +/- 56ms 2023-09-30T00:54:51.382 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 +723us[ +723us] +/- 76ms 2023-09-30T00:54:51.382 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:54:51.383 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:54:51.383 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2023-09-30T00:54:51.383 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 -261us[ -261us] +/- 91ms 2023-09-30T00:54:51.383 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -341us[ -331us] +/- 56ms 2023-09-30T00:54:51.384 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 40 +731us[ +731us] +/- 76ms 2023-09-30T00:54:51.384 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:54:51.384 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T00:54:51.394 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T00:54:51.394 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T00:54:51.403 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T00:54:51.535 DEBUG:teuthology.orchestra.run.smithi135:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T00:54:51.626 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:54:51.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:54:51.667 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:54:51.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:54:51.700 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T00:54:51.709 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T00:54:51.717 INFO:teuthology.task.internal:Duration was 5429.201074 seconds 2023-09-30T00:54:51.718 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T00:54:51.730 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T00:54:51.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:54:51.734 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:54:51.786 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:54:51.791 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:54:52.007 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T00:54:52.008 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2023-09-30T00:54:52.008 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:54:52.037 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi135.front.sepia.ceph.com 2023-09-30T00:54:52.038 DEBUG:teuthology.orchestra.run.smithi135:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:54:52.061 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T00:54:52.062 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:54:52.081 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:54:52.141 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T00:54:52.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:54:52.144 DEBUG:teuthology.orchestra.run.smithi135:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:54:52.247 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T00:54:52.256 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T00:54:52.256 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:54:52.291 DEBUG:teuthology.orchestra.run.smithi135:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:54:52.319 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T00:54:52.331 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:54:52.333 DEBUG:teuthology.orchestra.run.smithi135:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:54:52.365 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2023-09-30T00:54:52.386 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = core 2023-09-30T00:54:52.404 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:54:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:54:52.448 DEBUG:teuthology.orchestra.run.smithi135:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:54:52.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:54:52.465 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T00:54:52.475 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T00:54:52.476 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586/remote/smithi040 2023-09-30T00:54:52.476 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:54:52.534 DEBUG:teuthology.misc:Transferring archived files from smithi135:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586/remote/smithi135 2023-09-30T00:54:52.535 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:54:52.575 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T00:54:52.576 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:54:52.578 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:54:52.627 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T00:54:52.636 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T00:54:52.637 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T00:54:52.713 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T00:54:52.714 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:54:52.717 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:54:52.735 INFO:teuthology.orchestra.run.smithi135.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:54 /home/ubuntu/cephtest 2023-09-30T00:54:52.737 INFO:teuthology.orchestra.run.smithi040.stdout: 525658 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:54 /home/ubuntu/cephtest 2023-09-30T00:54:52.739 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T00:54:52.752 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T00:54:52.774 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T00:54:52.801 DEBUG:teuthology.nuke:shortname: smithi040 2023-09-30T00:54:52.801 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:54:52.829 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 23:15:47.011478', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPvPERG99uKqCE5cUKrNqI7eYuJHBXWPBlFgCuZx0NmMh3J9k7ndvs1VBB1bF9I+97RdCkmCFnsnYhpADHvH5Gs='} 2023-09-30T00:54:52.833 DEBUG:teuthology.nuke:shortname: smithi135 2023-09-30T00:54:52.833 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:54:52.855 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406586', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 23:15:47.013443', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPUUjOeXF2ooqKQhpZAj7iSD5ij/F0ffRzx+l37fx47VQbEGqYBo7TgFF0BSKZOzRhalCw7yOEhzZs87TAF2ueo='} 2023-09-30T00:54:52.885 INFO:teuthology.orchestra.console.smithi040:Power off 2023-09-30T00:54:52.885 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:54:52.913 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2023-09-30T00:54:52.913 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:54:52.935 INFO:teuthology.orchestra.console.smithi135:Power off 2023-09-30T00:54:52.936 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:54:52.946 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-09-30T00:54:52.960 DEBUG:teuthology.orchestra.console.smithi135:power off output: Chassis Power Control: Down/Off 2023-09-30T00:54:52.961 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:54:52.990 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2023-09-30T00:54:56.948 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:54:56.991 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:54:57.062 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2023-09-30T00:54:57.102 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2023-09-30T00:55:01.065 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:55:01.104 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:55:01.180 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2023-09-30T00:55:01.214 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is off 2023-09-30T00:55:01.282 INFO:teuthology.orchestra.console.smithi040:Power off completed 2023-09-30T00:55:01.315 INFO:teuthology.orchestra.console.smithi135:Power off completed 2023-09-30T00:55:01.420 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 5429.201074361801 failure_reason: '"1696034833.9469364 mon.a (mon.0) 70 : cluster [ERR] Health check failed: Module ''pg_autoscaler'' has failed: ''PgAutoscaler'' object has no attribute ''noautoscale'' (MGR_MODULE_ERROR)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=64c9917d687e4242bccecdb58f4172b0 status: fail success: false 2023-09-30T00:55:01.421 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:55:01.506 INFO:teuthology.run:FAIL